{"id":"6e44c6d9-d01b-4654-83cd-9c3b517c65ef","arxiv_id":"2412.01096","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The proposed FS-SNS feature selection method reportedly improves simulation accuracy for 8 of 10 real-world networks, but the evaluation is circular because the same degree-distribution metric is optimized and then used for selection.","lead":"This paper proposes FS-SNS, a feature selection method for social network simulators that ranks real-world node features and tests feature combinations. It claims improved degree distribution accuracy for 8 of 10 networks, but the evaluation is confounded because the same metric drives optimization and feature selection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is circular: the same degree-distribution JS divergence is used for both HyperOpt parameter tuning and feature selection, so the reported 8/10 accuracy gains are in-sample fits and do not establish that FS-SNS improves generalization.","rationale":"The reader's weakest-assumption pinpointed the same issue I consider most load-bearing: the reuse of the degree-distribution JS divergence as both the HyperOpt objective and the feature-selection criterion makes the reported accuracy gains in-sample fits. This is the central problem because it directly invalidates the headline claim that FS-SNS 'improves' simulation accuracy. Other weaknesses—the ad hoc down-sampling, the lack of error bars, and the weakly supported four-feature threshold—would weaken confidence in the results even if the evaluation were honest, but the circular evaluation severs the link between the reported numbers and the claim. A held-out or alternative-metric test would resolve whether the selected features carry genuine signal. I agree with the reader that rejection is appropriate: the current evidence cannot support the paper's central conclusion, and a meaningful revision would need to demonstrate out-of-sample or at least metric-independent improvement.","tokens_in":14276,"tokens_out":5961,"duration_ms":52928,"concrete_test":"Run the complete FS-SNS pipeline for each network exactly as described, but evaluate the final selected features and the Base SNS on a metric not used by either HyperOpt or the feature wrapper—for example, the Jensen-Shannon divergence of the clustering coefficient distribution (or held-out edge prediction). If FS-SNS does not outperform Base SNS on this held-out metric across the same 10 networks, the reported degree-distribution improvements are consistent with in-sample selection bias rather than a genuine increase in simulation accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FS-SNS improves simulation accuracy for 8 of 10 networks rests on an evaluation that is vulnerable to selection bias. In Section III-A, the degree distribution is 'chosen as the metric to optimise and evaluate the SNS', and HyperOpt minimizes the Jensen-Shannon divergence of the degree distribution to tune the pDNA/hDNA weights. In Section III-B, Algorithm 1 selects features using the same JS divergence: the wrapper adds ranked features one by one and stops when the 'degree distribution error computed using the Jensen–Shannon divergence' does not improve, and the feature combination with the error 'closest to 0' is chosen. Both stages therefore minimize the same objective on the same target network. This means the final reported error for FS-SNS is the training error of the whole selection-and-optimisation procedure, not an unbiased estimate of the method's accuracy. The comparison against Base SNS is biased because FS-SNS has the additional freedom to choose a feature subset that best fits the training metric; even uninformative features could reduce the training error through this search, and the absence of any held-out validation or alternative evaluation metric prevents distinguishing genuine improvement from overfitting. Consequently, the claimed 24% average improvement is not established. The 'four-feature threshold' claim inherits the same weakness, as it is derived from these same in-sample selections.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FS-SNS, a hybrid feature-selection method for a social network simulator (SNS) that ranks real-world node features using unsupervised filter metrics and then wraps a forward search over ranked feature subsets. The sDNA weights (pDNA and hDNA) are optimized with HyperOpt, and the feature subset is chosen by a wrapper that evaluates degree-distribution similarity via Jensen–Shannon divergence. The authors report that FS-SNS improves the degree-distribution error for 8 of 10 real-world networks, with an average improvement of 24%, and they also claim a consistent four-feature threshold for accurate simulation. Efficiency comparisons show that FS-SNS is slower than the Base SNS but, in the authors' view, worth the accuracy gain.","tokens_in":14552,"tokens_out":6280,"duration_ms":57162,"significance":"If the reported improvements were properly validated, the paper would be a useful practical contribution to complex network simulation and digital twin research: it uses publicly available datasets, provides a GitHub repository, and addresses the underexplored question of selecting real-world node features for SNSs. However, the current evaluation does not support the central claim. The feature-selection wrapper and the HyperOpt optimizer both minimize the same in-sample Jensen–Shannon divergence of the degree distribution, so the reported gains are training-set fits rather than evidence of genuine predictive improvement. The lack of any held-out evaluation, repeated runs, or error bars further weakens the empirical claims. The manuscript's main value is therefore the method proposal and the reproducible pipeline, not the quantitative conclusions as currently stated.","major_comments":[{"comment":"The evaluation is circular. Section III-A states that the degree distribution is 'the metric to optimise and evaluate the SNS,' and HyperOpt minimizes the Jensen–Shannon divergence of the degree distribution to tune the pDNA and hDNA weights. Section III-B then uses the same JS divergence in the wrapper: Algorithm 1 stops when the degree-distribution error does not improve, and the feature combination 'closest to 0' is selected. Thus the final FS-SNS error is obtained by minimizing the same objective on the same target network at both the weight-optimization and feature-selection stages. The comparison against Base SNS is biased because FS-SNS has the additional freedom to choose a feature subset that best fits the training metric. The reported 8-of-10 improvement and the 24% average improvement are therefore in-sample results and do not establish that FS-SNS generalizes. A proper evaluation would require a held-out set of nodes or edges, a separate evaluation metric, or at least a nested cross-validation-style protocol where feature selection and weight optimization are performed on training data and the final accuracy is measured on validation data.","section":""},{"comment":"The Twitch and Deezer networks are heavily downsampled before simulation, but the results are presented as simulations of 'Twitch' and 'Deezer.' For Twitch, the graph is reduced until it has only 1,000 edges, and for Deezer the number of nodes is cut roughly in half; the target degree distribution is then the distribution of the subsample, not of the original network. This is especially consequential because Deezer is one of the networks with the largest reported improvement (50%). Without validating that the sampling procedure preserves the properties relevant to the simulation, the results for these two networks cannot be taken as evidence about the full networks. The paper should either simulate the full networks, if computationally feasible, or explicitly reframe the claims and validate the sampling method (e.g., by comparing several samples and reporting variance).","section":""},{"comment":"All results come from single runs with fixed random seeds (seed 50 for network formation and seed 42 for HyperOpt). No repeated trials, confidence intervals, or significance tests are reported. The claim that FS-SNS 'improves' 8 of 10 networks is therefore vulnerable to run-to-run variation; the accuracy differences in Table III, which are as small as 0.01 for the Bison and Songbird networks, could easily be within the noise of the stochastic simulation and optimization procedure. The authors should repeat the experiments over multiple seeds or initializations and report means and variances, and ideally a paired statistical test over the ten networks.","section":""},{"comment":"Algorithm 1 is inconsistent with the surrounding text. The text says the goal is to minimize the Jensen–Shannon divergence error, i.e., values closer to zero are better. In the pseudocode, however, the variables 'M DD' and 'GM DD' denote a 'Max Degree Distribution' and the update condition 'if DD > M DD' selects larger values as better. Moreover, the pseudocode line 'Set pDNA and hDNA' does not state that these weights are obtained from the HyperOpt optimization described in Section III-A, which is an essential step of the method. As written, the algorithm is not reproducible and does not match the experimental description.","section":""},{"comment":"The claimed 'threshold of 4 features' is not supported by the evidence. Table III shows that the selected feature counts are 1, 1, 2, 4, 1, 1, 2, 1, 4, and 1; 4 is simply the maximum number of features selected in any network. This does not establish a threshold, and the statement that '4 or fewer features' achieve the most accurate simulation is trivially true given the selection procedure. If the authors intend to claim that no network benefits from more than four features, they need to test combinations beyond four features and show that accuracy does not improve.","section":""}],"minor_comments":[{"comment":"Table I says the data sets are 'shown in the appendix,' but no appendix is present in the manuscript; the table itself already provides the network names and sizes, so the reference to the appendix should be removed or the appendix should be added.","section":""},{"comment":"Equation (1) is ambiguous: the notation P R(1), P R(2), P R(3) is not defined, and the fraction '/3' appears to be misplaced. Please rewrite the formula with explicit definitions of all symbols.","section":""},{"comment":"The caption of Figure 5 mentions a 'red line' for FS-SNS, but the figure as provided has two curves without clear color labels; please add a legend and ensure the color description matches the rendered figure.","section":""},{"comment":"The naming of feature-ranking methods is inconsistent: 'FR-Var-Col-Lap' appears in Table III, while the discussion refers to 'FS Col Lap' and 'FR-Var-Col-Lap' interchangeably. Please use one consistent terminology.","section":""},{"comment":"Typographical errors: 'random see' should be 'random seed' in two places, and 'node pari scores' should be 'node pair scores.'","section":""},{"comment":"Several references have incomplete author information (e.g., [7], [10], [16]) or placeholder initials; please complete them according to the citation style.","section":""}],"recommendation":"major_revision","confidential_remarks":"The circularity concern raised in the reader's report is real and is the central issue: the same degree-distribution JS divergence is used for weight optimization, feature selection, and final evaluation, so the reported improvements are in-sample. I would be willing to consider a revised version that adds a proper validation protocol (e.g., held-out evaluation, separate metric, or repeated trials) and addresses the downsampling issue. As it stands, the quantitative claims are not supported, although the method itself is a plausible contribution worth testing properly. The paper may also benefit from tightening the narrative around the 'four-feature threshold' and improving the reproducibility of Algorithm 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a reasonable engineering case study. It applies standard unsupervised filters (variance, multicollinearity, Laplacian score, mutual information) plus a wrapper to an existing social network simulator, and it ships code and data. The per-network discussion of why certain features matter for baboons, wolves, or ants is thoughtful, and the efficiency numbers are honestly reported even though FS-SNS is slower. Those parts are worth a look.\n\nThe central claim, however, is not established. The same Jensen–Shannon divergence of the degree distribution is used as the objective for HyperOpt parameter tuning and as the criterion for the feature wrapper. Wrapper Algorithm 1 adds ranked features and keeps the combination with lowest error on that same metric. So the reported 8/10 improvement over Base SNS is essentially training error after an additional selection step; with the extra freedom to pick a subset, FS-SNS can only do better or tie in sample. The average 24% improvement is a fit statistic, not a fair measure of simulation accuracy, and the claimed four-feature threshold inherits the same problem. There is no held-out evaluation, no repeated runs, and no error bars. The down-sampling for Twitch and Deezer is ad hoc and could distort the degree distributions being matched. A random-feature-selection baseline would have helped show the filter rankings actually contribute something beyond the wrapper search.\n\nNone of this means the paper is worthless. The pipeline is clearly described, the code is available, and the qualitative findings about feature informativeness are plausible. But the quantitative headline needs a validation scheme that separates parameter fitting from feature selection, for example by using a different metric for selection than for reporting, or by evaluating on a held-out subgraph.\n\nMy recommendation: this deserves serious peer review rather than a desk rejection, but only with major revisions. The core result has to be rebuilt around out-of-sample evaluation before the accuracy claims can be taken seriously.","headline":"A clear engineering write-up of feature selection for a social network simulator, but the headline 8/10 accuracy gain is an in-sample artifact because the same degree-distribution error drives both parameter tuning and feature selection.","tokens_in":15041,"tokens_out":2421,"would_cite":false,"duration_ms":25531,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feature selection lifts network simulation accuracy by 24 percent on average.","keywords":["feature selection","complex network simulation","social network simulation","degree distribution","Jensen-Shannon divergence","unsupervised filtering","node heterogeneity","digital twin"],"falsifier":"Take one of the ten networks, split its node set into a fitting subset and a reserved subset, run FS-SNS using only the fitting subset to choose features, then measure the Jensen-Shannon divergence on the reserved subset; if the chosen features do not beat the full-feature baseline on the reserved subset in most networks, the claim that FS-SNS improves simulation accuracy is not established.","tokens_in":14093,"feed_emoji":"🕸️","tokens_out":3840,"duration_ms":30866,"temperature":0.7,"pith_summary":"The paper asks whether machine-learning-style feature selection can improve simulations of real-world social networks. It proposes FS-SNS, a hybrid method that first ranks each node's real-world attributes with four unsupervised filter scores and then adds attributes one at a time, in rank order, to the simulator while monitoring the Jensen-Shannon divergence between the simulated and target degree distributions. Across ten real-world networks, FS-SNS lowers the degree-distribution error for eight of them, with an average accuracy improvement of 24 percent and a best case of 69 percent. The authors also report that four or fewer features sufficed to produce the most accurate simulation for every network tested, and that the added computational cost is modest compared with the accuracy gain.","feed_headline":"Feature selection lifts network simulation accuracy by 24 percent","feed_subtitle":"A rank-and-wrap method improves 8 of 10 social network simulations, using four or fewer node features.","key_machinery":"The load-bearing mechanism is a hybrid feature-selection wrapper. It computes four unsupervised filter scores for each node attribute: variance, multicollinearity, Laplacian score, and mutual information; sums them with a weighting that rewards attributes frequently ranked first, second, or third; and then feeds the ranked list to a wrapper that adds one feature at a time to the base social network simulator. After each addition the simulator's sDNA weights are re-optimised with HyperOpt, and the degree distribution of the resulting network is compared with the target using Jensen-Shannon divergence. The wrapper stops when adding the next feature no longer improves that divergence, so the final feature set is the one whose simulation is closest to the target among all combinations tested. This makes the feature subset, rather than any single score, the unit that carries the accuracy gain.","core_discovery":"The central claim is that including real-world node features in a social network simulator, chosen by a rank-then-wrap feature selection procedure, yields materially more accurate simulations than the same simulator using all available features. The discovery is empirical: on the ten cleaned real-world networks, the FS-SNS algorithm improved the degree-distribution error in 8 out of 10 cases, reduced it by an average of 24 percent, and achieved its best combination with four or fewer features on every network. The method's name is FS-SNS, for feature selection with social network simulations. The result is presented as a step toward digital twins of real-world social systems, because it shows which node attributes actually carry information about connection patterns.","pith_inferences":["A natural next test is to check whether the four-feature threshold generalises to larger or differently structured networks, since the ten datasets here are small and largely animal or media networks.","Because the wrapper stops on in-sample divergence, applying the same selection rule with a held-out network or a different objective, such as clustering coefficient, would show whether the ranking reflects genuine signal or optimisation artefacts.","The same rank-and-wrap logic could be applied to edge and topological features, not just node attributes, which the authors suggest as future work.","The finding that a single top-ranked feature often beats all features suggests a simple diagnostic for any network simulator: before adding complexity, test each attribute alone."],"forward_implications":["Using FS-SNS instead of the Base SNS reduces degree-distribution error for 8 of the 10 tested real-world networks, with an average improvement of 24 percent.","The most accurate simulation for every tested network uses four or fewer selected features, so small feature subsets can replace the full attribute set.","Networks whose connection patterns follow preferential attachment and heterogeneity, such as online social platforms and dominance-structured animal groups, benefit most from feature selection.","The efficiency cost of FS-SNS is an extra optimisation run per feature tested; for the wolf network the accuracy gain reaches 69 percent for a modest time increase.","The feature ranking step can be applied as a systematic diagnostic for which node attributes matter, even before committing to a full simulation run."],"supporting_citations":[{"why":"Defines the base social network simulator with sDNA weights that FS-SNS extends.","marker":"[12]"},{"why":"Establishes the degree-distribution evaluation and the prior SNS architecture used as the baseline.","marker":"[13]"},{"why":"Earlier study on heterogeneous feature representation that motivates including real-world node features.","marker":"[24]"},{"why":"HyperOpt package used to optimise sDNA weights for each feature combination.","marker":"[25]"},{"why":"Supplies the Jensen-Shannon divergence used as the accuracy metric and wrapper stop criterion.","marker":"[28]"},{"why":"Animal Social Network Repository, source of several datasets.","marker":"[29]"},{"why":"SNAP datasets source for Twitch and Deezer networks.","marker":"[30]"},{"why":"UCINET datasets source for wolf and other networks.","marker":"[31]"},{"why":"Review of feature selection methods that motivates the filter and wrapper taxonomy.","marker":"[7]"},{"why":"Laplacian score method used in the filtering step.","marker":"[19]"}],"fun_headline_variants":["Rank-and-wrap feature selection boosts network sim accuracy 24%","FS-SNS: 4 features pick lifts simulation accuracy by 24%","Feature selection improves 8 in 10 network simulations","Fewer node features, 24% better network simulation accuracy","Rank-then-wrap picks 4 features, cuts simulation error 24%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that degree-distribution error measured on the same network used for optimisation is a trustworthy guide for choosing features; because the wrapper selects features using the same Jensen-Shannon divergence that HyperOpt optimises, the reported gains are in-sample fits rather than predictions on unseen networks.","fun_headline_variants_meta":{"raw":{"variants":["Rank-and-wrap feature selection boosts network sim accuracy 24%","FS-SNS: 4 features pick lifts simulation accuracy by 24%","Feature selection improves 8 in 10 network simulations","Fewer node features, 24% better network simulation accuracy","Rank-then-wrap picks 4 features, cuts simulation error 24%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1444,"prompt_tokens":849,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":505}},"tokens_in":465,"tokens_out":595,"duration_ms":5556,"temperature":1.0,"reasoning_tokens":505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:40:48.963906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the ten networks, split its node set into a fitting subset and a reserved subset, run FS-SNS using only the fitting subset to choose features, then measure the Jensen-Shannon divergence on the reserved subset; if the chosen features do not beat the full-feature baseline on the reserved subset in most networks, the claim that FS-SNS improves simulation accuracy is not established.","supporting_citations":[{"cited_title":"Simulation and Augmentation of Social Networks for Building Deep Learning Models","cited_arxiv_id":"1905.09087","evidence_quote":"Defines the base social network simulator with sDNA weights that FS-SNS extends."},{"cited_title":"Digital Twin-Oriented Complex Networked Systems based on Heterogeneous Node Features and Interaction Rules","cited_arxiv_id":"2308.11034","evidence_quote":"Establishes the degree-distribution evaluation and the prior SNS architecture used as the baseline."},{"cited_title":"Heterogeneous Feature Representation for Digital Twin-Oriented Complex Networked Systems","cited_arxiv_id":"2309.13229","evidence_quote":"Earlier study on heterogeneous feature representation that motivates including real-world node features."},{"cited_title":"Hyperopt: a python library for model selection and hyperparameter optimization,","cited_arxiv_id":null,"evidence_quote":"HyperOpt package used to optimise sDNA weights for each feature combination."},{"cited_title":"Variability analysis of complex networks measures based on stochastic distances,","cited_arxiv_id":null,"evidence_quote":"Supplies the Jensen-Shannon divergence used as the accuracy metric and wrapper stop criterion."},{"cited_title":"A multi-species respository of social networks,","cited_arxiv_id":null,"evidence_quote":"Animal Social Network Repository, source of several datasets."},{"cited_title":"Borgatti, M","cited_arxiv_id":null,"evidence_quote":"UCINET datasets source for wolf and other networks."},{"cited_title":"A review of feature selection methods with applications,","cited_arxiv_id":null,"evidence_quote":"Review of feature selection methods that motivates the filter and wrapper taxonomy."},{"cited_title":"Iterative laplacian score for feature selection,","cited_arxiv_id":null,"evidence_quote":"Laplacian score method used in the filtering step."}],"review_version":1}