{"id":"28f60084-bb81-4bab-b4e5-5c9ea2296ff3","arxiv_id":"2607.06528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"A trust-aware pipeline combining SciBERT citation-intent classification with six structural features ranks suspicious citation communities on a 500K-paper DBLP graph, surfacing communities with 254x citation inflation and 64% superficial citations.","lead":"The paper builds a pipeline that combines citation graph structure with LLM-labeled citation intent to rank suspicious citation communities in scholarly networks. A smart generalist might read it to understand how semantic signals can improve detection of coordinated citation manipulation beyond topology-only methods.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The edge excision experiment (§3.1) is circular: CCI includes density and inflation by construction, so top-ranked communities are necessarily dense, and removing their internal edges will always differ from random removal — this validates density, not CCI's trust-aware design.","rationale":"The reader correctly identified three concerns: no ground-truth validation, uncalibrated trust weights, and edge excision circularity. I agree with all three but would reorder them: the edge excision circularity is the most load-bearing because it undermines a key experimental claim (RQ3) that is presented as independent validation. The trust-weight concern is real but partially mitigated by the ablation results showing ranking robustness. The ground-truth concern is important but the paper is honest about framing output as an audit queue, so it doesn't undermine the paper's stated goals as directly. The CONDITIONAL verdict remains appropriate: the pipeline is a legitimate scientometrics contribution, but the edge excision experiment as currently designed cannot distinguish CCI's value from density-based selection. Adding the density/inflation baseline excision comparison I propose would either confirm or refute the circularity concern definitively. The reader's MODERATE confidence is appropriate given the missing supplementary details (LLM prompt, SciBERT hyperparameters).","tokens_in":7109,"tokens_out":3027,"duration_ms":55890,"concrete_test":"Re-run the edge excision experiment using communities selected by the density-only and inflation-only baselines (Table 3 variants) instead of full CCI. Specifically: identify the top-5 communities under density-only ranking, remove all their internal edges, and measure giant component retention. If the retention rate is within ~1 percentage point of CCI's 94.1%, then the excision result is a trivial consequence of density selection, not evidence for CCI's trust-aware discriminative power. If CCI-selected communities show meaningfully lower retention (e.g., <92%), the semantic features do add graph-level signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's RQ3 asks whether highly ranked communities exhibit graph-level behavior differing from matched random edge removals. The edge excision removes all 404,625 internal edges from the top-5 CCI communities and compares against random removals, finding 94.1% vs 99.95–99.96% giant component retention (§3.1). But CCI includes internal directed density (feature 1) and citation inflation (feature 2) by construction (§2.3, Appendix B). Top-ranked communities are therefore necessarily dense with many internal edges. Removing concentrated edges from dense communities will always produce a different graph-level signature than removing the same number of edges uniformly at random — this is a property of any density-based community selection, not specific to CCI's trust-aware features. The experiment would only validate CCI's added value if it compared excision of CCI-top communities against excision of communities selected by density-only or inflation-only baselines. If those baselines produce similar excision effects, then the semantic features contribute nothing to the graph-level claim, and RQ3 is answered trivially. This is more load-bearing than the trust-weight concern (the reader's primary worry) because the ablation in Table 4 already shows the ranking is somewhat robust to feature removal (Spearman 0.875–0.886 when semantic features are dropped), whereas the excision experiment has no such control against structural-only selection. The reader identified this as concern #3 but did not treat it as the primary weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper presents a trust-aware pipeline for ranking suspicious citation communities in scholarly knowledge graphs. The approach combines six structural and semantic features into a Composite Cartel Index (CCI): internal density, citation inflation, reciprocity, semantic superficiality, degree assortativity, and trust-weighted PageRank shift. Citation intent is scaled via an LLM teacher (205,897 labeled pairs) distilled into a SciBERT student, applied to 2.04M edges of a 500K-paper DBLP-derived graph. The authors validate CCI against single-heuristic baselines, leave-one-feature ablations, multi-seed stability, and an edge-excision stress test. The top-ranked community shows 254.3x citation inflation and 64.2% superficial citations. The framing as a curator-facing audit queue rather than a misconduct detector is appropriate.","tokens_in":7426,"tokens_out":1218,"duration_ms":117840,"significance":"The paper addresses a genuine gap: combining citation-intent semantics with structural anomaly detection at paper-level community granularity, rather than journal- or author-level. The teacher-student pipeline scaling to 2.04M typed edges is a practical engineering contribution. The reproducibility artifacts (teacher labels, semantic edges, scores, ablations) and the falsifiable baseline comparisons in Tables 3-4 are commendable. The six-feature CCI is interpretable and the ablation framework is well-structured. However, the significance is tempered by two issues: the absence of any ground-truth validation confirming that top-ranked communities are actual cartels, and the circularity risk in the edge-excision experiment (RQ3), which I discuss below.","major_comments":[{"comment":"§3.1, Edge Excision Experiment (RQ3): The edge excision experiment is partly circular. CCI includes internal directed density (feature 1) and citation inflation (feature 2) by construction (§2.3, Appendix B). Top-ranked communities are therefore necessarily dense with many internal edges, and removing concentrated edges from dense communities will always produce a different graph-level signature than uniform random removal. The experiment validates density concentration, not CCI's trust-aware design. To validate the added value of the semantic features, the excision should compare CCI-top communities against communities selected by density-only or inflation-only baselines. If those baselines produce similar excision effects, the semantic features contribute nothing to the graph-level claim, and RQ3 is answered trivially. This is load-bearing because RQ3 is one of three stated research q ","section":null},{"comment":"§2.3, Table 1: The six semantic trust weights (Method=1.0, Result/Comparison=0.7, Support=0.5, Contrast/Criticism=0.3, Background=0.2, Perfunctory/Ceremonial=0.1) are hand-set without empirical calibration or sensitivity analysis. These weights propagate through two of six CCI features (trust-weighted PageRank shift and semantic superficiality). Table 4 shows these two features have the largest impact on top-five membership changes when removed (PageRank drop: P@5 drops to 0.60; semantic superficiality: P@10 drops to 0.70). A sensitivity analysis showing ranking stability under perturbed weights (e.g., ±20% on each weight) would substantially strengthen the claim that the ranking is robust. Without it, the specific numeric choices remain unjustified.","section":null},{"comment":"§2.2 and §3: No ground-truth validation is provided. The paper never confirms that top-ranked communities are actual cartels—only that they score high on CCI and behave differently from random removals. While the authors correctly frame CCI as an audit queue rather than a definitive detector, even a small manual inspection of the top-5 communities (e.g., checking author overlap, self-citation rates, or institutional ties) would substantially strengthen the claim that the ranking surfaces genuine anomalies rather than legitimate dense research communities. The 254.3x inflation and 64.2% superficiality for the top community are suggestive but not confirmatory.","section":null}],"minor_comments":[{"comment":"§2.2: The SciBERT macro-F1 of 0.574 is low, and the paper acknowledges rare classes are difficult. However, the Perfunctory/Ceremonial F1 of 0.473 is particularly relevant since this class feeds directly into the semantic superficiality feature. A brief discussion of how classification errors propagate into community-level superficiality estimates would help.","section":null},{"comment":"Table 2: The CCI values drop sharply from rank 1 (1.863) to rank 2 (0.441). This suggests the top community is an extreme outlier. A brief comment on whether this is expected or concerning would help interpretation.","section":null},{"comment":"§2.1: The snowball sampling from the 200 most-cited papers introduces a potential selection bias. A sentence acknowledging this limitation would be appropriate.","section":null},{"comment":"Reference [5] is listed as 'Anonymous. 2025. Citation Intent. Preprint.' This should be properly attributed or removed.","section":null},{"comment":"Figure 6 is referenced but not shown in the text provided; ensure it clearly labels the CCI-excision vs. random-excision curves.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about circularity in the edge excision experiment is well-founded and is the most important issue. The reader's concern about trust weights is also valid but secondary, since the ablation already shows some robustness. The lack of any ground-truth validation is the third leg: without at least a small manual check, the paper cannot confirm that its ranking surfaces actual cartels rather than legitimate dense communities. All three issues are fixable within the manuscript's scope (add baseline-controlled excision, add sensitivity analysis, add manual inspection of top-5), hence major revision rather than reject."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee correctly identifies two substantive weaknesses—lack of ground-truth validation and circularity risk in the edge-excision experiment—both of which we will address in revision. We also agree that sensitivity analysis on the semantic trust weights is needed. Below we respond point by point.","responses":[{"response":"The referee is correct that the current excision experiment does not isolate the contribution of the semantic features. As stated, RQ3 validates that CCI-selected communities form concentrated local structures with a distinct graph signature, but it does not establish that the trust-aware components are responsible for this distinctiveness rather than density alone. We will revise the experiment to include the suggested comparison: excising edges from communities selected by density-only and inflation-only baselines and comparing the graph-level impact against CCI-selected communities. If the baselines produce similar excision effects, we will state this plainly and reframe RQ3 accordingly. If CCI-selected communities show a different signature, this would support the trust-aware design. Either outcome is informative and we will report it honestly. We will also revise the text to acknowledge the circularity concern in the current version.","revision_made":"yes","referee_comment":"§3.1, Edge Excision Experiment (RQ3): The edge excision experiment is partly circular. CCI includes internal directed density (feature 1) and citation inflation (feature 2) by construction. Top-ranked communities are therefore necessarily dense, and removing concentrated edges from dense communities will always produce a different graph-level signature than uniform random removal. To validate the added value of the semantic features, the excision should compare CCI-top communities against communities selected by density-only or inflation-only baselines."},{"response":"We agree. The weights in Table 1 are motivated by the rhetorical distinction between substantive and superficial citations, but they are not empirically calibrated, and the referee is right that this matters given that the two trust-weighted features show the largest impact in the ablation (Table 4). We will add a sensitivity analysis perturbing each weight by ±20% (and possibly a broader range) and report the effect on top-k ranking stability, Spearman correlation with the original ranking, and top-five membership changes. If the ranking is unstable under perturbation, we will report that transparently and discuss implications for the robustness of the trust-aware signals.","revision_made":"yes","referee_comment":"§2.3, Table 1: The six semantic trust weights are hand-set without empirical calibration or sensitivity analysis. These weights propagate through two of six CCI features. A sensitivity analysis showing ranking stability under perturbed weights would substantially strengthen the claim that the ranking is robust."},{"response":"This is a fair and important criticism. We deliberately framed CCI as a curator-facing audit queue rather than a definitive detector, but the referee is right that even a small manual inspection would strengthen the paper substantially. We will conduct a manual review of the top-5 communities, examining author overlap, self-citation rates, institutional ties, and publication venue patterns. We will report findings honestly: if the inspection reveals indicators consistent with coordinated citation behavior, this supports the ranking; if the communities appear to be legitimate dense research communities, we will state this and discuss what it means for the approach. We acknowledge that without this validation, the current claim that CCI surfaces genuine anomalies is not fully supported.","revision_made":"yes","referee_comment":"§2.2 and §3: No ground-truth validation is provided. The paper never confirms that top-ranked communities are actual cartels. Even a small manual inspection of the top-5 communities would substantially strengthen the claim that the ranking surfaces genuine anomalies rather than legitimate dense research communities."}],"tokens_in":6796,"tokens_out":774,"duration_ms":179557,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"This paper combines LLM-distilled citation intent classification with six-feature structural-semantic community scoring to rank suspicious citation communities in a 500K-paper DBLP graph. The engineering is real: 205K teacher-labeled citation pairs, a SciBERT student scaled to 2M typed edges, and a composite index (CCI) that integrates density, inflation, reciprocity, semantic superficiality, assortativity, and trust-weighted PageRank shift. The ablations in Tables 3–4 are the strongest part — they show CCI is not reducible to any single heuristic (density-only Spearman 0.513, structural-only 0.875), and that removing semantic features shifts top-five membership. The paper is also appropriately modest in framing the output as an audit queue, not a verdict. That framing is correct and I give them credit for it. The graph profiling in Section 2.1 is competent and motivates the feature choices reasonably. The teacher-student pipeline is a legitimate scaling contribution. Now the soft spots. The biggest one is the edge excision experiment (§3.1), and the stress-test note lands here. CCI includes internal density and citation inflation as features by construction, so top-ranked communities are necessarily dense. Removing 405K concentrated internal edges from dense communities and observing different giant-component behavior than uniform random removal is expected for any density-based selection — it does not validate the trust-aware features specifically. The experiment would only support the paper's claim if it compared excision of CCI-top communities against excision of density-only-top or inflation-only-top communities. Without that control, RQ3 is answered trivially. The reader flagged this as concern #3; I think it is the most load-bearing weakness, more important than the trust-weight issue. The trust weights (Table 1) are hand-set without sensitivity analysis, and they propagate through two of six features. Table 4 shows PageRank drop and semantic superficiality are the features whose removal most affects top-five membership, so the weights matter. A perturbation analysis would be straightforward and is needed. The SciBERT macro-F1 of 0.574 is low but defensible for aggregate community-level scoring — the paper makes this argument and it is reasonable, though a human spot-check of even 50 top-ranked community edges would strengthen it. The 42% edge typing coverage is acceptable but worth noting. No ground-truth cartel validation exists. The paper never confirms that the top-ranked community (254x inflation, 64% superficiality) is actually a cartel. This is the gap between audit queue and validated tool. The paper is honest about this, but it limits the claims. This is for scientometrics researchers and scholarly-graph curators. It deserves a serious referee — the pipeline is novel at this scale, the ablations are informative, and the framing is responsible. The referee should push hard on the excision control and the weight sensitivity analysis. would_accept_peer_review: true","headline":"Solid engineering of a trust-aware citation cartel ranking pipeline at meaningful scale, but the central validation experiment is partly circular and there is no ground-truth confirmation that flagged communities are actually cartels.","tokens_in":7877,"tokens_out":1062,"would_cite":false,"duration_ms":23212,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Citation Cartels Caught by Combining What Citations Say With How They Cluster","keywords":[],"falsifier":"If perturbing the trust weights within a reasonable range causes the top-ranked communities to change substantially, or if the communities flagged by CCI turn out (under expert audit) to be legitimate dense research communities rather than suspicious ones, the ranking's practical value would be undermined.","tokens_in":7067,"feed_emoji":"🔍","tokens_out":897,"duration_ms":191345,"temperature":0.7,"pith_summary":"The paper argues that citation cartels in scholarly knowledge graphs can be reliably surfaced for audit only when two signals are combined: the graph-structural pattern of a community (excessive internal density, reciprocity, inflation beyond degree-product expectation, assortativity) and the semantic quality of the citations within it (whether they represent substantive method use or shallow ceremonial background). The authors construct a pipeline on a 500K-paper DBLP subgraph: an LLM teacher labels citation intent on ~206K edges, a SciBERT student scales intent classification to 2.04M edges, and six community-level features are averaged into a single ranking score called the Composite Cartel Index (CCI). The central claim is that CCI cannot be reduced to any single heuristic—density alone correlates only 0.51 with the final ranking, and semantic-only or inflation-only baselines each miss or misrank communities—and that top-ranked communities exhibit a distinct graph-level signature under edge excision (removing their internal edges destabilizes the giant component far more than matched random removals). The headline finding is a community of 1,079 papers with 254x citation inflation and 64.2% superficial internal citations.","feed_headline":"254x Citation Inflation Surfaced by Trust-Aware Graph Ranking","feed_subtitle":"Combining citation intent semantics with graph structure flags suspicious communities that no single heuristic can isolate, on a 500K-paper,","key_machinery":"Composite Cartel Index (CCI): a community-level score computed as the mean of six z-scored features—internal density, citation inflation ratio, reciprocity, semantic superficiality fraction, degree assortativity, and PageRank drop from unweighted to trust-weighted computation. Citation intent is classified into six categories (Method, Result/Comparison, Support, Contrast/Criticism, Background, Perfunctory/Ceremonial) by a SciBERT student trained on LLM-labeled pairs, then mapped to trust weights ranging from 1.0 to 0.1.","core_discovery":"The core discovery is that suspicious citation communities occupy a region of feature space that no single structural or semantic signal isolates. Density, inflation, reciprocity, assortativity, semantic superficiality, and trust-weighted PageRank shift each capture a different failure mode, and their combination surfaces communities that are simultaneously hyper-dense, semantically shallow, reciprocally reinforcing, and peripherally positioned in the global citation backbone. The edge-excision experiment confirms these are not statistical artifacts: removing 405K internal edges from the top-five CCI communities removes 5.8 percentage points more nodes from the giant component than randomrem","pith_inferences":[],"forward_implications":["Citation-graph platforms could deploy CCI-style ranking as a triage layer for human curators, surfacing compact communities with disproportionate internal citation and shallow semantic support before they distort aggregate impact metrics.","The finding that semantic superficiality and PageRank drop are the features whose removal most affects top-five membership suggests that intent-aware citation weighting is not decorative—it changes which communities get flagged.","If the six trust weights were empirically calibrated against adjudicated cartel cases rather than hand-set, the ranking could become both more defensible and more sensitive to cartel strategies that adapt to known weightings.","Edge excision as a validation paradigm could be applied to other graph-anomaly ranking systems: if removing top-ranked anomalies does not produce a distinct global structural signature, the ranking may be surfacing noise rather than coordinated behavior."],"fun_headline_variants":["Cartel detection requires six signals, not just citation density","64% of citations in top-ranked cartel community are superficial","Trust-weighted PageRank shift exposes citation cartels","Composite Cartel Index isolates communities no single heuristic catches","Semantic citation intent separates cartels from legitimate clusters"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The six semantic trust weights (Method=1.0 down to Perfunctory/Ceremonial=0.1) are set by hand without empirical calibration or sensitivity analysis, yet they propagate through two of the six CCI features and the ablation shows those two features are the ones whose removal most changes the top-five ranking.","fun_headline_variants_meta":{"raw":{"variants":["Cartel detection requires six signals, not just citation density","64% of citations in top-ranked cartel community are superficial","Trust-weighted PageRank shift exposes citation cartels","Composite Cartel Index isolates communities no single heuristic catches","Semantic citation intent separates cartels from legitimate clusters"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":697,"prompt_tokens":620,"completion_tokens":77,"prompt_tokens_details":null},"tokens_in":620,"tokens_out":77,"duration_ms":36671,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T02:44:05.483269+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If perturbing the trust weights within a reasonable range causes the top-ranked communities to change substantially, or if the communities flagged by CCI turn out (under expert audit) to be legitimate dense research communities rather than suspicious ones, the ranking's practical value would be undermined.","supporting_citations":[],"review_version":1}