{"id":"f893e3af-caa4-4501-ada9-806656de1578","arxiv_id":"1908.08715","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GImpact measures pairwise scientific influence between institutions from grant records via heat diffusion on institution and discipline/keyword graphs, then clusters institutions by those influence scores.","lead":"This paper presents GImpact, a graph-based service that scores how Chinese universities influence each other through shared research grants, using heat-diffusion formulas over institution, discipline, and keyword networks. It also clusters institutions into collaboration groups and tests the approach on a 14-year grant dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation metrics as written undermine the effectiveness claim: Eq.(30) counts all intra-cluster vertex pairs rather than collaboration edges, and Eq.(32)'s scatter term inherits diagonal scores N+1 that grow with the number of combined influence graphs, biasing the reported DBI comparison.","rationale":"The paper's graph-theoretic pipeline is coherent: heat-diffusion kernels, co-influence similarity, and a k-medoids variant are all reasonable building blocks, and the grant-graph construction is a plausible service-oriented contribution. The central claim, however, is empirical: that the method 'effectively' identifies grant-based collaboration clusters. For that claim to hold, the reported density and DBI comparisons must measure what they claim. Eq.(30) as printed lacks the edge-membership predicate and therefore counts arbitrary vertex pairs inside clusters, so it is not a valid graph-cohesiveness metric. Eq.(31) includes centroid self-scores in sigma_j while Eq.(17) defines those self-scores as N+1; DBI comparisons across settings with different numbers of influence graphs are then confounded by construction. Both defects are concrete, checkable, and fixable, which is why I treat this as a verification failure rather than evidence that the method is wrong. The reader's circularity concern is related but not identical: DBI is self-referential in roughly the expected way, but the density problem is more basic, since the formula as written does not even reference the edge set. With no released code or data, no external labels, and no baseline clustering algorithm, the current manuscript does not establish its headline effectiveness claim. A corrected evaluation could change that, so I recommend UNVERDICTED pending the concrete rerun described above.","tokens_in":25251,"tokens_out":12478,"duration_ms":141734,"concrete_test":"Recompute the two evaluation metrics exactly as the text intends: (i) density with the edge predicate restored, sum_j |{(iv_m,iv_n) in IE : iv_m,iv_n in I_j}| / |IE|, and (ii) DBI with sigma_j computed only over iv_l in I_j with iv_l != cv_j. Rerun Figures 5a and 5b for K = 25, 50, 100, 200 using the Table 7 weights. If the combined-graph advantage over 'Institution only' shrinks or reverses, the effectiveness claim is unsupported; if the advantage survives both corrections, report the corrected numbers and the cluster size distributions.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim that GImpact can 'effectively identify' collaboration groups (Section 6.3) rests almost entirely on the density and DBI comparisons. As printed, Eq.(30) is sum_j |{(iv_m,iv_n) | iv_m,iv_n in I_j}| / |IE|. The numerator has no (iv_m,iv_n) in IE predicate, so it counts every pair of institutions that falls in the same cluster, whether or not they share any joint grant. That measures cluster-size balance, not collaboration cohesion, and the reported density advantage of the combined model could be an artifact of uneven cluster sizes. The surrounding text says density should reflect 'current institution collaboration relationships,' so either the equation is misstated or the implementation computes something different; in either case the published result is not reproducible as written. Second, Eq.(31) defines sigma_j as the average S(cv_j, iv_l) over all iv_l in I_j without excluding the centroid. Since Eq.(17) sets the diagonal of the overall score to N+1, each sigma_j contains a self-similarity term whose magnitude grows with the number of aspect graphs. The 'Institution+Discipline+Keyword' setting (N=2, diagonal=3) is compared against 'Institution' (N=0, diagonal=1) under different self-score contributions, so the reported DBI improvement may be a construction artifact. Because no external ground truth, no baseline clustering method, and no released code or data are provided, the effectiveness evidence collapses until these metric definitions are corrected and rerun.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GImpact, a graph-based framework for analyzing scientific influence in a research grants repository. It constructs an institution collaboration graph plus two aspect graphs (disciplines and keywords), computes self-influence scores by heat diffusion on the institution graph and co-influence scores by heterogeneous heat diffusion on institution-aspect graphs, and combines them into an overall pairwise influence score S = S0(α) + Σωi Si. It then performs K-medoids clustering using these scores as distances, with custom centroid initialization, assignment, and update schemes, and it proposes hierarchical-structure-based optimizations (supplement and aggregate connections). The evaluation uses a Chinese MOE humanities/social science grant dataset (2512 institutions, 334,068 grants, 2005–2018) and reports density and Davies-Bouldin Index (DBI) comparisons across configurations, initialization/assignment/update schemes, and optimization strategies. The paper claims that GImpact can effectively identify grant-based research collaboration groups and provide insight into institutional influence.","tokens_in":25633,"tokens_out":3024,"duration_ms":31562,"significance":"If the effectiveness claim were supported, GImpact would be a useful contribution: it applies a heat-diffusion influence model to grant data, combines direct and indirect collaboration signals across multiple aspect graphs, and introduces practical optimizations for hierarchical disciplines. The problem is timely and the graph-based framework is coherent. However, the current evidence for effectiveness is weak: the two quantitative metrics in Section 6.2 are computed from the same pairwise influence matrix that the clustering algorithm is designed to optimize, and one metric (density, Eq. (30)) as written does not even measure collaboration edges. The case study (Tables 8–13) is illustrative but not a validation. The paper would be significantly strengthened by corrected metrics, external or at least independent validation, and a principled specification of free parameters. Given the soundness of the core model, these issues are fixable within the manuscript's scope.","major_comments":[{"comment":"The density metric as printed counts all pairs of institutions that co-occur in a cluster: the numerator is sum_j |{(iv_m,iv_n) | iv_m, iv_n in I_j}| with no condition (iv_m,iv_n) ∈ IE. This measures cluster-size balance, not collaboration cohesion, so the reported density advantage of the combined model in Figs. 5–9 could be an artifact of uneven cluster sizes. The surrounding text says density should reflect 'current institution collaboration relationships,' so either the equation is misstated or the implementation computes something different; in either case the published result is not reproducible as written. Please correct the formula to count actual collaboration edges within clusters (or justify and rename the metric).","section":"§6.2, Eq. (30)"},{"comment":"The DBI calculation is biased across the compared configurations. Eq. (31) defines σ_j as the average S(cv_j, iv_l) over all iv_l in I_j without excluding the centroid, and Eq. (17) sets the diagonal of S to N+1, where N is the number of aspect graphs. Thus each σ_j includes a self-similarity term that grows with N. Comparing 'Institution' (N=0, diagonal=1) against 'Institution+Discipline+Keyword' (N=2, diagonal=3) mixes this construction artifact into the reported DBI improvements in Fig. 5b. Please exclude the diagonal from σ_j or otherwise normalize the configuration-wise comparison.","section":"§6.2, Eqs. (31)–(32)"},{"comment":"The weights α, ω1, ω2 are set per K with no stated selection rule, and they vary substantially across K (e.g., α ranges from 0.15 to 2.2). Since the overall influence score in Eq. (16) and every downstream clustering result depend on these weights, the effectiveness comparisons across configurations are not interpretable without either a principled selection criterion (e.g., a validation set or sensitivity analysis) or an explicit statement that the reported weights are hand-tuned. Please provide a selection rule and/or a sensitivity analysis.","section":"§6.1, Table 7"},{"comment":"The supplement coefficients λ1 and λ2 are introduced in Eq. (27) but their values are never given in the experiments. The claim that λ2 should be larger than λ1 is qualitative. Without concrete values (or a procedure for choosing them), the 'Supplement' and 'Aggregation+Supplement' results in Fig. 9 cannot be reproduced. Please report the parameter values and, if they are tuned, describe how they were chosen.","section":"§5.2, Eq. (27); §6.5"},{"comment":"The effectiveness claim rests entirely on internal metrics (density and DBI) that are computed from the same pairwise influence matrix S that the GImpact clustering algorithm is designed to maximize. The paper provides no external ground truth, no expert labels, and no baseline clustering method (e.g., Louvain, spectral clustering, or standard K-medoids with adjacency-based distances). The case study in §6.6 is illustrative but does not constitute validation. Please add at least one external or independent check, or a comparison against a non-gradient baseline, so that the claim 'can effectively identify the grant-based research collaboration groups' is not self-confirmatory.","section":"§6.3–§6.5"}],"minor_comments":[{"comment":"The text says 'The associated weights for different K we used in the experiments list at Figure 7.' This should refer to Table 7, not Figure 7.","section":"§6.1"},{"comment":"Panels (j) and (l) both appear to show the same 'Overall Influence Score' matrix; if they are meant to be different stages or views, the labels should be differentiated.","section":"Figure 2"},{"comment":"The row label '# of Discipline' should be '# of Disciplines' for grammatical consistency with the other rows.","section":"Table 5"},{"comment":"The phrase 'combining both quantify (the number of publications) and quality' should read 'quantity'.","section":"§2.2"},{"comment":"The norm used in ‖s_k − s_j‖ is not specified; please state whether it is Euclidean or another norm.","section":"§4.4.1, Eq. (24)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core graph-theoretic model is internally consistent, but the evaluation section is the main weakness. The density and DBI issues are specific and fixable, and the lack of external validation is the largest risk. If the authors can correct the metrics, specify all parameters, and add a baseline or independent validation, the paper could be publishable. I would not recommend rejection solely on the current self-referential validation, because the proposed method is novel and the underlying optimization is coherent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:1908.08715. The core of the paper is a coherent framework for grant influence analysis: build an institution graph plus discipline and keyword aspect graphs, compute self-influence via heat diffusion and co-influence via a heterogeneous diffusion kernel, combine them with a weighted sum, and cluster with a medoids variant that has some sensible heuristics. The diffusion kernels are explicitly borrowed from [17] and [18], and the new pieces — the grant-specific graph construction, the co-influence pattern similarity Eq. (15), and the integration with a custom clustering objective — are reasonable. The dataset is real and sizable (2512 institutions, 334k grants). So the modeling contribution is useful, and the paper could be a good basis for a research analytics service.\n\nThe problem is the evaluation. The two metrics used to support the central claim don't measure what the text says. Eq. (30) defines density as the total number of vertex pairs inside clusters divided by |IE|. That has no collaboration edge predicate; it just grows with cluster sizes. So a clustering that puts all vertices in one giant cluster would score high, which is not \"cohesiveness\" or \"collaboration relationships.\" Either the equation is misstated or the implementation does something different, but as printed it's not a valid measure. Eq. (32) has a subtler but equally serious issue: σ_j as defined includes the centroid's self-score S(cv_j, cv_j), which is N+1 by Eq. (17). So when you compare Institution-only (N=0, diagonal 1) to Institution+Discipline+Keyword (N=2, diagonal 3), the combined model gets a systematically lower DBI just because of the larger diagonal term. That's a construction artifact, not evidence.\n\nThere are also free parameters everywhere: the weights α and ω_i in Table 7 are set independently for each K with no selection rule, λ1 and λ2 in Eq. (27) are never given values, and there's no sensitivity analysis. No external ground truth, no baseline clustering method, no code or data. So the paper's headline claim that GImpact 'can effectively identify' collaboration groups is unsupported by the evidence as presented.\n\nAll of this is fixable. Correct the density to count intra-cluster edges, define σ_j excluding the centroid, do a systematic sweep over weights, and add at least one external validation (e.g., known institutional groupings or co-publication data). If the authors do that, the contribution would be solid. As it stands, I'd treat the effectiveness claim as not established.\n\nWho is this for? People working on grant analytics or heterogeneous network clustering will find the modeling ideas useful. I'd bring it to a reading group as a cautionary example of self-referential evaluation, but not as a definitive method. I would not cite it as evidence for the effectiveness of the approach. But it does deserve peer review, because the underlying framework is sound and the evaluation can be repaired. Recommend: send it to review, but flag the metric problems clearly and ask for a reanalysis.","headline":"A coherent model on borrowed kernels, but the evaluation metrics as printed undercut the central effectiveness claim.","tokens_in":26122,"tokens_out":5314,"would_cite":false,"duration_ms":50384,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-theoretic service computes pair-wise scientific influence scores between institutions from grant records alone, and clusters those scores into collaboration groups the raw data do not make visible.","keywords":["research grants","scientific influence analysis","graph mining","heat diffusion","heterogeneous information network","co-influence score","K-medoids clustering","collaboration detection"],"falsifier":"One could test whether the clusters reflect genuine collaboration by comparing them against an independent signal the method never sees, such as co-authored publications or formal partnership agreements between the same institutions, with institution size and geographic distance controlled for. If GImpact's clusters match this external ground truth no better than clusters built from raw joint-grant counts or from a standard community-detection baseline, then the reported density and DBI improvements would not establish that the influence scores capture scientific influence. A simpler test is to shuffle discipline and keyword assignments across grants and re-run the clustering; if clusters barely change, the co-influence terms are not carrying the claimed information.","tokens_in":24993,"feed_emoji":"📊","tokens_out":5784,"duration_ms":57956,"temperature":0.7,"pith_summary":"This paper tries to establish that a graph-theoretic analysis service, GImpact, can turn a raw research-grant database into pair-wise \"scientific influence\" scores between institutions and that clustering those scores surfaces collaboration groups that the grant records alone do not make visible. It claims that combining a heat-diffusion self-influence score over joint-grant collaborations with co-influence scores routed through discipline and keyword graphs captures both direct and indirect relationships between institutions. The authors evaluate the approach on a real dataset of 2512 institutions over 14 years and report that the resulting clusters have higher density and lower DBI than clusters built from institution-only scores. If this is right, grant repositories could support influence analysis without relying on publication citations.","feed_headline":"Grant data alone can expose hidden research collaboration clusters","feed_subtitle":"A weighted graph score from joint grants, disciplines, and keywords ranks 2,512 institutions.","key_machinery":"The central machinery is the multi-hop heat-diffusion influence kernel $e^{{αtH}}$ (Eq. 7), where H is a one-hop influence-spread kernel whose off-diagonal entries are normalized edge weights. On the institution graph this kernel yields the self-influence matrix S0; on each heterogeneous institution-aspect graph, a block version of H encodes institution-institution, aspect-aspect, and cross edges, and the institution-to-aspect part of $e^{{αtH}}$ gives each institution's co-influence pattern vector. The co-influence score between two institutions is the product of the cosine similarity and a length-ratio term of their pattern vectors (Eq. 15), and the overall score is the weighted sum in Eq. (16). This overall score supplies the similarity input to a customized K-medoids clustering algorithm that uses mixed centroid initialization, dynamic assignment, and a max-objective centroid update.","core_discovery":"The paper's central claim is that the overall scientific influence score S(m,n) between any two institutions, defined by Eq. (16) as S = S0(α) + Σ ωi Si, integrates three kinds of evidence: direct joint grants, indirect reachability through the institution graph, and the shared or related disciplines and keywords attached to grants. The self-influence term S0 comes from applying a heat-diffusion kernel to the institution graph, and each co-influence term Si is the similarity of the institutions' influence patterns on a discipline or keyword graph. With this score used as a distance metric in a K-medoids clustering algorithm, GImpact claims to \"effectively identify the grant-based research collaboration groups\" and to offer insight into institution leadership and future collaboration opportunities.","pith_inferences":["A time-stamped generalization is natural: weighting recent joint grants more heavily than older ones could turn the static influence matrix into a time series that tracks the rise and fall of collaboration groups; the paper does not consider temporal weighting.","The symmetric positive score matrix invites cross-validation with spectral or modularity-based clustering; if those methods recover the same groups, the result is unlikely to be an artifact of the specific centroid-update scheme.","If the effectiveness claim holds, grant-based influence scores could be used as a predictive feature for future joint-grant applications or co-authorships, an outcome the paper does not test.","The same heterogeneous influence machinery transfers to other record-based networks, such as patents, clinical trials, or procurement contracts, where an institution-like entity connects to category-like attributes."],"forward_implications":["If the claims hold, research administrators can rank prospective partners by grant-based influence score in a repository where grant records are the only structured data.","The clustering output can reveal research-area-defined groups and identify institutions that lead particular subject areas, purely from funding records.","Because the framework is attribute-generic, the same self-influence and co-influence pipeline can be pointed at individual researchers or departments instead of institutions, as the paper notes.","The hierarchical-discipline optimization, which both supplements and aggregates connections, shows a way to scale influence computation on repositories that have hierarchical taxonomies."],"supporting_citations":[{"why":"Supplies the heat-diffusion model for influence spread that underlies the self-influence score.","marker":"[17]"},{"why":"Supplies the heterogeneous influence-spread model and the pattern-vector approach for co-influence scores.","marker":"[18]"},{"why":"Supplies the K-medoids clustering algorithm that GImpact extends with new initialization, assignment, and centroid-update schemes.","marker":"[19]"},{"why":"Inspires the mixed centroid initialization scheme that uses average and maximum influence scores to separate initial centroids.","marker":"[21]"},{"why":"Provides the cluster-separation metric used to evaluate clustering quality.","marker":"[23]"}],"fun_headline_variants":["Grant graphs reveal hidden collaboration clusters","GImpact: scoring institutions by joint grants and disciplines","Influence scores from grant data rank 2,512 institutions","Graph-theoretic grant analysis exposes research clusters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the cluster-quality metrics (density and DBI) computed from the very same pairwise influence score S that the clustering algorithm optimizes are unbiased evidence that the resulting groups are scientifically meaningful, with no external ground truth or independent baseline.","fun_headline_variants_meta":{"raw":{"variants":["Grant graphs reveal hidden collaboration clusters","GImpact: scoring institutions by joint grants and disciplines","Influence scores from grant data rank 2,512 institutions","Graph-theoretic grant analysis exposes research clusters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1243,"prompt_tokens":975,"completion_tokens":268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":209}},"tokens_in":591,"tokens_out":268,"duration_ms":3266,"temperature":1.0,"reasoning_tokens":209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:31:16.391944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One could test whether the clusters reflect genuine collaboration by comparing them against an independent signal the method never sees, such as co-authored publications or formal partnership agreements between the same institutions, with institution size and geographic distance controlled for. If GImpact's clusters match this external ground truth no better than clusters built from raw joint-grant counts or from a standard community-detection baseline, then the reported density and DBI improvements would not establish that the influence scores capture scientific influence. A simpler test is to shuffle discipline and keyword assignments across grants and re-run the clustering; if clusters barely change, the co-influence terms are not carrying the claimed information.","supporting_citations":[{"cited_title":"Mining social networks using heat diffusion processes for marketing candidates selection,","cited_arxiv_id":null,"evidence_quote":"Supplies the heat-diffusion model for influence spread that underlies the self-influence score."},{"cited_title":"Social influence based clustering of hetero- geneous information networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the heterogeneous influence-spread model and the pattern-vector approach for co-influence scores."},{"cited_title":"Kaufman and P","cited_arxiv_id":null,"evidence_quote":"Supplies the K-medoids clustering algorithm that GImpact extends with new initialization, assignment, and centroid-update schemes."},{"cited_title":"k-means++: The advantages of careful seeding,","cited_arxiv_id":null,"evidence_quote":"Inspires the mixed centroid initialization scheme that uses average and maximum influence scores to separate initial centroids."},{"cited_title":"A cluster separation measure,","cited_arxiv_id":null,"evidence_quote":"Provides the cluster-separation metric used to evaluate clustering quality."}],"review_version":1}