{"id":"c0f121c1-b2bc-420a-b33b-4ab7b58a6bca","arxiv_id":"2412.14867","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A document clustering pipeline that builds a named-entity similarity graph, then runs graph convolutional clustering on LLM embeddings, outperforms co-occurrence and KNN graph baselines in the paper's experiments.","lead":"The paper combines named entity recognition with large language model embeddings inside a graph convolutional network to cluster news and scientific documents. It reports large gains over word-frequency-based clustering on four datasets, especially when documents share named entities.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.4 selects the number of clusters k using ground-truth labels (explicitly for Arxiv-10), inflating the reported ACC/NMI/ARI and undermining the unsupervised superiority claim of GCC*.","rationale":"The paper's strongest claim is that GCC* outperforms all comparison methods, including KNN-graph clustering. The evaluation in Section 4.4 selects the number of clusters k after inspecting dendrograms and, for Arxiv-10, explicitly picks 10 'to remain consistent with the benchmarks.' This is a clear use of ground-truth labels to set a key hyperparameter, and it affects the reported ACC, NMI, and ARI for every method. Although all baselines are given the same k, the paper's unsupervised framing makes this label leakage a direct threat to the claim: if k is not chosen data-driven, the method is not 'unsupervised' in the sense claimed, and the absolute scores are inflated. The reader identified this as the weakest assumption, and I agree. Other concerns, such as inconsistent threshold notation and missing error bars, are secondary to the fact that the core comparison may be biased by label-informed k. The proposed concrete test would resolve whether the relative advantage survives a truly unsupervised k selection.","tokens_in":10505,"tokens_out":6788,"duration_ms":55721,"concrete_test":"Re-run GCC* and the strongest baseline (GCC on Aknn with LLM embeddings) on Arxiv-10 and BBC News, selecting k without any label information: use an internal criterion such as the eigengap of the NER graph Laplacian or silhouette maximization over the GCC* embedding YpW, and do not consult ground truth. Report ACC, NMI, and ARI for the label-free k. If the margin between GCC* and the baseline narrows or reverses on Arxiv-10, the headline advantage is an artifact of label-informed k selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 describes choosing k by running GCC* with 250–500 clusters, applying Ward hierarchical clustering on the centroids, and inspecting dendrograms. For Arxiv-10, the authors write: 'several partitions can be considered, with 3, 4, or 10 clusters. To remain consistent with the benchmarks and our evaluation study of GCC*, we opted for 10 clusters.' This uses the known class count to set k, which is a free parameter in the objective (Eqs. 7–9) and directly determines the partition. The same label-guided dendrogram inspection is used for all four datasets. Because ACC, NMI, and ARI all depend on k, the reported numbers in Table 2 are inflated relative to a genuinely unsupervised selection. Since the paper's abstract and Section 4.4 explicitly claim an unsupervised framework, this label leakage means the central claim that GCC* is an unsupervised method outperforming baselines is not established by the experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes GCC*, a document-clustering pipeline that modifies the GCC joint embedding/clustering method of [11] in two ways: it replaces bag-of-words features with LLM embeddings (OpenAI text-embedding-3-small) as the node feature matrix Xℓℓm, and it replaces the k-NN adjacency with a named-entity graph Aner, in which two documents are connected when they share enough entities whose Word2Vec embeddings have cosine similarity above a threshold, with edge weights given by Eq. (4). The clustering objective (Eqs. (7)-(9)), the λ=1 setting, and the alternating optimization are taken from [11]. The authors report ACC, NMI, and ARI on four corpora (BBC News, a 16,321-document five-category subset of MLSUM, PubMed, and a 10,000-document sample of Arxiv-10), claiming that GCC* on (Aner, Xℓℓm) exceeds all evaluated baselines, including KNN-graph GCC, co-occurrence-based deep clustering, and k-means on LLM embeddings, and they illustrate the learned embeddings with UMAP.","tokens_in":10679,"tokens_out":14587,"duration_ms":103211,"significance":"If the empirical claim were established, the contribution would be a simple, low-cost recipe for injecting entity-level relational structure into LLM-based document clustering, with a transparent pipeline (NER extraction, Word2Vec entity similarities, thresholded document graph, GCC optimization) that is easy to replicate from Algorithm 1, and a benchmark sweep that includes a French corpus and a scientific-abstract corpus, which is a strength. The paper is also candid about its protocol choices, which is what makes the evaluation problems identifiable. However, the optimization machinery is entirely inherited from [11], so the novelty is the adjacency construction and the feature choice; the central claim is therefore an empirical one, and it currently rests on an evaluation protocol that selects k with knowledge of the true labels and reports no variance or significance measures. The contribution is promising but not yet demonstrated.","major_comments":[{"comment":"The choice of the number of clusters k leaks the ground truth into the evaluation. The text states for Arxiv-10 that 'several partitions can be considered, with 3, 4, or 10 clusters. To remain consistent with the benchmarks and our evaluation study of GCC∗, we opted for 10 clusters,' and for the other datasets that 'the true number of clusters was easily identified,' meaning the dendrogram inspection is calibrated against the benchmark labels. Because k is a free parameter of the objective in Eqs. (7)-(9) and directly determines the partition, and because ACC, NMI, and ARI in Table 2 are computed against those same labels, the reported gains (e.g., BBC News NMI 95.12%) conflate method quality with oracle knowledge of k. This contradicts the paper's own framing ('Since our framework is unsupervised') and the abstract's unsupervised-superiority claim. The same paragraph selects p only 'once the number of classes has been set,' so the p=2/3 choice inherits the leakage. A clean protocol would either fix k to the class count for all methods and separately evaluate k-detection without labels, or report Table 2 across a range of k values; either way, a label-free k-selection experiment (e.g., against an internal criterion or on a held-out split) is required before the headline claim can be accepted.","section":"Section 4.4"},{"comment":"The reported means over 20 runs come without standard deviations or significance tests, which matters because several headline comparisons are small. On PubMed, GCC∗ (ACC 65.10, NMI 29.04, ARI 25.00) is within 0.10 pp of GCC(knn, Xℓℓm) on ACC (65.00) and exactly tied on ARI (25.00); on Arxiv-10 the ACC gap is 2.87 pp. Without a paired test (e.g., Wilcoxon over the 20 runs) or variance reporting, the claim that GCC∗ 'exceeds all other methods' is not quantitatively supported on these datasets. Please report mean ± std and significance levels for at least the main comparison rows.","section":"Table 2"},{"comment":"The threshold τ is overloaded to mean two different things. In Section 3.3.3 and Section 4.3 ('we only kept links where τ was greater than 0.9'), τ is a cosine-similarity cutoff between entity embeddings, whereas in Section 3.3.4 and Eq. (4) ('the number of shared entities Edi ∩ Edj is greater than or equal to a predefined threshold τ') τ is the minimum count of shared entities, and Algorithm 1 uses τ for the similarity comparison while hardcoding 'at least 3 common named entities.' Since Eq. (4) is the definition of the adjacency matrix Aner on which the entire method rests, the construction is ambiguous: a reader cannot determine from the paper whether the count threshold is τ (as in Eq. (4)) or 3 (as in Algorithm 1 and Section 4.3). Please introduce distinct names (e.g., τ_sim for the cosine cutoff and m for the minimum number of shared entities), and reconcile Eq. (4), Algorithm 1, and the experimental setting accordingly.","section":"Sections 3.3.3-3.3.4, Eq. (4), Algorithm 1"},{"comment":"The evaluation corpus is not the one described in Table 1. Table 1 reports MLSUM as 407,835 documents with 612 clusters, but Section 4.3 says that 'we randomly sample 10,000 Arxiv documents and 16,321 MLSUM documents from five categories' (Sport, Health, Politics, Economy, Climate); the MLSUM results in Table 2 therefore appear to be for a 16,321-document, 5-class subset, not for the corpus in Table 1, and the effective k for MLSUM is left ambiguous (5 vs 612). Similarly, the Arxiv-10 column is presumably the 10,000-document sample, yet Table 1 gives 100,000 documents. The paper should state the actual number of documents and classes used for evaluation, present the subset statistics, and clarify which k was used for each column of Table 2.","section":"Table 1 and Section 4.3"}],"minor_comments":[{"comment":"The text cites 'Figure 4' for the KNN-vs-NER graph comparison, but that figure is Figure 1; Figure 4 is the UMAP figure in Section 4.5. The cross-reference should be corrected.","section":"Introduction"},{"comment":"The text says the edge weight is 'proportional to the similarity of the named entities' without saying whether aij is the sum or the average of the pairwise entity similarities; also, since entities are matched across documents by cosine similarity rather than string equality, the intersection Edi ∩ Edj in Eq. (4) and the count constraint should be defined over matched entity pairs, not string-identical tokens.","section":"Eq. (4)"},{"comment":"The equivalence of (8) and (9) is asserted by reference to [2, 3, 17, 11] without a derivation; please provide the derivation for this objective form with Agg = T^p Xℓℓm, or cite the precise result in [11] that covers it.","section":"Section 3.4, Eqs. (8)-(9)"},{"comment":"The abstract's 'notably for documents rich in named entities' is not tested, as no experiment varies entity density across documents; please qualify the claim or add an ablation that varies entity density.","section":"Abstract and Section 4.5"},{"comment":"Decimal formatting is inconsistent (e.g., 55.8 vs 55.80, 0.9 vs 0.90, 0.0), and the table should state explicitly that all entries are percentages.","section":"Table 2"},{"comment":"The comparison set omits the graph-clustering and NER-clustering methods reviewed in Section 2 (e.g., DGI, MVGRL, GraphCL, BERT-Kmeans, and [6, 8]); 'exceeds all other methods' should be scoped to the evaluated baselines.","section":"Section 2 vs Section 4"},{"comment":"The caption says the UMAP embeddings were 'obtained with GPT,' but the paper elsewhere states that embeddings were generated with OpenAI's text-embedding-3-small model; please use the embedding model name consistently.","section":"Section 4.3 and Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the incremental contribution over the same group's GCC [11] and LLM-clustering [13] work is real but thin, since the optimization, the (8)=(9) equivalence, and the λ=1 setting are all inherited from [11]; the novelty lives in the NER-based adjacency and the LLM features. The Section 4.4 k-selection admission is the decisive issue: it is an explicit use of the benchmark labels to set a free parameter of the objective, so the Table 2 headline numbers overstate the method's unsupervised performance. I recommend requiring a label-free k-selection or k-sensitivity analysis and paired significance tests before this can be considered for publication. The citation density to the authors' own prior work ([1], [2], [3], [11], [13], [17], [27]) is high, though the citations are topically relevant."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Keraghel and Nadif propose GCC*, which swaps the usual KNN adjacency for a graph built from named-entity overlap and similarity, then feeds it to their earlier GCC joint embedding/clustering objective with LLM embeddings. The NER graph construction (Section 3.3) is the real new piece: train Word2Vec on the corpus, embed named entities, and connect documents when they share enough similar entities. That is a sensible, practical idea, and the paper does a fair job describing the pipeline. The BBC News and MLSUM numbers are large and consistent, and the gain from LLM embeddings over BoW matches prior work.\n\nThe soft spots are serious. Section 4.4 chooses the number of clusters k by inspecting dendrograms and explicitly opting for 10 on Arxiv-10 'to remain consistent with the benchmarks' — i.e., with knowledge of the true class count. Since k is a free parameter in the objective and directly determines the partition, ACC/NMI/ARI in Table 2 are inflated relative to a genuinely unsupervised selection. The paper claims an unsupervised framework, but the headline comparison is not established on that basis. Relatedly, there are no standard deviations or significance tests despite 20 runs, and the propagation power p and the PC Kmeans 500-link constraint are tuned on the same data. The tau notation is also inconsistent: it is a similarity threshold in Step 3 but a count threshold in Step 4 and Algorithm 1. I would also note the heavy reliance on GPT-4o for NER and OpenAI embeddings, with no code or data released, which limits reproducibility.\n\nNone of this kills the underlying idea. The NER graph is a legitimate input that may genuinely help entity-rich corpora; the problem is that the current evaluation does not convincingly separate that effect from the label-informed k. The optimization being from the authors' own GCC is not itself a flaw — reusing your own method is fine — and the LLM-embedding-for-clustering work is incremental but clearly related.\n\nThis is a paper for a practitioner audience working on document clustering with entity-heavy text. A careful reader can extract the NER graph recipe, but should not cite Table 2 as evidence for unsupervised superiority. I would send it to peer review with a request for heavy revision: re-run with an unsupervised k (or a sensitivity range), report variance and significance, fix the tau notation, and release code and data. If the gains survive an honest k choice, it becomes a decent applied contribution.","headline":"A genuinely new NER-based graph construction is buried under an evaluation whose cluster count comes from the ground-truth labels, so the headline unsupervised gains are not yet established.","tokens_in":11232,"tokens_out":3615,"would_cite":false,"duration_ms":28226,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing co-occurrence or KNN edges with named-entity similarity edges, while keeping LLM embeddings as node features and jointly optimizing embeddings and clusters, yields better document clusters than the…","keywords":["Large Language Models","Named Entity Recognition","Graph Convolutional Networks","Node Embedding","Node Clustering","Document Clustering","LLM Embeddings","Named Entity Graph"],"falsifier":"Run the same pipeline with k selected by a fully unsupervised internal criterion, such as the largest silhouette score or the minimum Davies-Bouldin index over the same grid, or by a dendrogram cut made before the annotator sees the true labels; if the GCC* row no longer beats GCC with the KNN graph and LLM embeddings, and no longer beats LLM-only k-means, the central claim fails. A cheaper check is to re-run Arxiv-10 with k=3 or k=4, the other partitions the dendrogram suggests, and see whether the advantage over baselines survives.","tokens_in":10271,"feed_emoji":"📊","tokens_out":12411,"duration_ms":74668,"temperature":0.7,"pith_summary":"This paper tries to establish that document clustering improves when the graph that connects documents is built from shared named entities instead of word co-occurrence or k-nearest-neighbour lexical similarity. The authors pair that entity-similarity adjacency matrix with LLM embeddings as node features, then run a graph-convolutional clustering objective that learns embeddings and cluster assignments together. On four corpora (BBC News, MLSUM, PubMed, Arxiv-10), the resulting pipeline, GCC*, reports higher accuracy, normalized mutual information, and adjusted Rand index than co-occurrence-based, KNN-graph, and LLM-embedding-only baselines. The reason to care is that clustering entity-heavy documents such as news requires relational links between documents that surface word statistics miss.","feed_headline":"Named entities in the graph lift clustering above LLM-only baselines","feed_subtitle":"A graph of shared named entities raises clustering accuracy on news, scientific, and medical texts","key_machinery":"The load-bearing object is the named-entity adjacency matrix $A_{\\text{ner}}$. Its entries average the cosine similarities of same-type entity pairs between two documents, with an edge allowed only when at least $\\tau = 0.9$ similarity is exceeded and at least three shared entities connect the documents. This matrix is fed, together with LLM embeddings $X_{\\ell\\ell m}$ from a 1536-dimensional text-embedding model, into the GCC objective: $\\min \\|Y_p - Y_p W W^\\top\\|^2 + \\|Y_p W - G F\\|^2$, where $Y_p = T^p X_{\\ell\\ell m}$ and $T$ is the normalized simple-graph-convolution matrix with self-loops; the orthogonality-constrained problem reduces to $\\min \\|Y_p - G F W^\\top\\|^2$ and is solved by alternating updates of $G$, $F$, and $W$. The machinery's work is to let entity-level semantic relations shape the graph convolution, so the learned embeddings and cluster assignments are pulled toward the structure of named entities rather than surface word overlap.","core_discovery":"The central claim is that a document graph whose edges encode named-entity similarity, combined with LLM-derived document embeddings and a joint embedding-and-clustering objective, yields clusters that match ground-truth classes better than any of the components alone. Concretely, an edge between two documents exists only when they share at least three same-type named entities whose Word2Vec cosine similarity exceeds a threshold of 0.9, and the edge weight averages those similarities; this named-entity adjacency matrix replaces the usual KNN or co-occurrence adjacency. Feeding this matrix together with LLM embeddings into the GCC objective, minimizing reconstruction error plus a k-means-style clustering penalty on the embedded nodes, produces the reported gains, for example an accuracy of 97.61% with a normalized mutual information of 95.12% on BBC News and an NMI of 72.42% on MLSUM, exceeding the KNN-graph variant and every other baseline in each table row. The authors also observe that using the entity graph alone with spectral clustering fails because the graph is too sparse, and that the joint graph-convolutional step is what makes the entity graph usable.","pith_inferences":["A testable extension the paper leaves open: restricting edges to a single entity type, such as organizations, locations, or persons, may sharpen clusters for topic-specific corpora, since the current edge weight averages similarities across all same-type entities.","Because the number of clusters is chosen by inspecting dendrograms with the true class structure in view, and set to the true class count for Arxiv-10, the headline numbers likely represent an upper bound; a fair unsupervised comparison would fix k by an internal criterion, and only if the gains persist there would GCC* be a practical recipe for entity-rich clustering.","The linear-algebra equivalence used to justify the objective suggests the named-entity adjacency matrix could be plugged into other spectral or matrix-factorization clustering methods, but the paper only tests it inside GCC.","On PubMed the gain over LLM embeddings alone is small (65.10% versus 63.25% accuracy), so the method's advantage may concentrate in documents with many named entities, which is exactly the regime the abstract claims."],"forward_implications":["If GCC* is correct, then on entity-rich corpora a clustering pipeline should treat named entities as first-class relational evidence: on BBC News, accuracy rises from 95.82% with the KNN graph to 97.61% with the named-entity graph while using the same LLM embeddings.","The joint embedding-and-clustering step is necessary to unlock the entity graph: spectral clustering on the NER adjacency alone yields near-zero NMI on PubMed, whereas GCC* keeps the graph useful by learning cluster-friendly embeddings.","LLM embeddings provide the dominant baseline jump over bag-of-words representations, and the named-entity graph adds a smaller but consistent gain on top of them across all four datasets.","The construction is model-agnostic: with Word2Vec for entity vectors and different NER models for English and French, stronger future NER or embedding models could be substituted without changing the method."],"supporting_citations":[{"why":"It supplies the GCC joint embedding-and-clustering objective and the alternating optimization that GCC* adapts.","marker":"[11]"},{"why":"Word2Vec is the trained similarity model whose cosine similarities define the edge weights in the named-entity adjacency matrix.","marker":"[20]"},{"why":"It is the prior comparison establishing that LLM embeddings are effective clustering features, which motivates using X_llm instead of bag-of-words.","marker":"[13]"},{"why":"It provides the Simple Graph Convolution aggregation T^p used to propagate the graph signal in the encoder.","marker":"[40]"},{"why":"It is the DeBERTa-based NER model used to extract named entities for the English datasets.","marker":"[34]"},{"why":"Arxiv-10 is one of the four benchmark datasets on which GCC* is evaluated.","marker":"[10]"},{"why":"MLSUM is the French news benchmark used to show the method works across languages.","marker":"[30]"},{"why":"It justifies the one-hop linear graph autoencoder used in the clustering objective.","marker":"[28]"}],"fun_headline_variants":["NER-weighted graph edges lift clustering beyond LLM embeddings alone","Shared-entity graph with Word2Vec threshold beats KNN in clustering","Entity-rich documents cluster 97.6% accurate with NER-graph GCN","Clustering gains from named-entity edges and joint embedding loss","Named entity similarity graph outperforms LLM-only clustering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the number of clusters as chosen without using the true labels, but the choice is made by inspecting dendrograms against the known classes and is set to the true class count for Arxiv-10; label leakage here would inflate every reported gain.","fun_headline_variants_meta":{"raw":{"variants":["NER-weighted graph edges lift clustering beyond LLM embeddings alone","Shared-entity graph with Word2Vec threshold beats KNN in clustering","Entity-rich documents cluster 97.6% accurate with NER-graph GCN","Clustering gains from named-entity edges and joint embedding loss","Named entity similarity graph outperforms LLM-only clustering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000534,"raw_usage":{"total_tokens":2544,"prompt_tokens":897,"completion_tokens":1647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1557}},"tokens_in":513,"tokens_out":1647,"duration_ms":8662,"temperature":1.0,"reasoning_tokens":1557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:50:04.317572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with k selected by a fully unsupervised internal criterion, such as the largest silhouette score or the minimum Davies-Bouldin index over the same grid, or by a dendrogram cut made before the annotator sees the true labels; if the GCC* row no longer beats GCC with the KNN graph and LLM embeddings, and no longer beats LLM-only k-means, the central claim fails. A cheaper check is to re-run Arxiv-10 with k=3 or k=4, the other partitions the dendrogram suggests, and see whether the advantage over baselines survives.","supporting_citations":[{"cited_title":"Efficient graph convolution for joint node representation learning and clustering","cited_arxiv_id":null,"evidence_quote":"It supplies the GCC joint embedding-and-clustering objective and the alternating optimization that GCC* adapts."},{"cited_title":"Beyond words: a comparative analysis of llm embeddings for effective clustering","cited_arxiv_id":null,"evidence_quote":"It is the prior comparison establishing that LLM embeddings are effective clustering features, which motivates using X_llm instead of bag-of-words."},{"cited_title":"Simplifying graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"It provides the Simple Graph Convolution aggregation T^p used to propagate the graph signal in the encoder."},{"cited_title":"T-NER: An all-round python library for transformer-based named entity recognition","cited_arxiv_id":null,"evidence_quote":"It is the DeBERTa-based NER model used to extract named entities for the English datasets."},{"cited_title":"Protoformer: Embedding prototypes for transformers","cited_arxiv_id":null,"evidence_quote":"Arxiv-10 is one of the four benchmark datasets on which GCC* is evaluated."},{"cited_title":"MLSUM: The Multilingual Summarization Corpus","cited_arxiv_id":"2004.14900","evidence_quote":"MLSUM is the French news benchmark used to show the method works across languages."},{"cited_title":"Simple and effective graph autoencoders with one-hop linear models","cited_arxiv_id":null,"evidence_quote":"It justifies the one-hop linear graph autoencoder used in the clustering objective."}],"review_version":1}