{"id":"936a10a1-4bb1-4035-bd88-496aed09b399","arxiv_id":"2505.07157","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HAMLET combines LLM-generated topics with graph-neural-network embedding refinement and reports better topic quality scores on English and French healthcare datasets than using unrefined embeddings.","lead":"HAMLET is a topic modeling pipeline that asks GPT-4o to propose topics for healthcare texts, then uses a graph neural network to refine those topic embeddings. The authors report improved topic coherence and stability on six datasets in English and French, but do not compare HAMLET against existing topic models on a shared metric.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ablation gain is partly built into the metric: Coherence_t scores topic embeddings against the very word nodes that EdgeRefineGNN aggregates into those embeddings during message passing, so §5.3's 0.740 vs 0.663 result is not independent evidence of semantic improvement.","rationale":"I read the paper as an unsupervised topic-labeling pipeline: GPT-4o generates candidate topic phrases, SBERT-BERT embeds them, SGS builds a heterogeneous graph, EdgeRefineGNN refines topic embeddings via edge-conditioned message passing, and K-means plus a coherence/centroid/connectivity selection produces the top k phrases. For the central claim—that refining embeddings improves topic quality—the load-bearing evidence is the ablation in §5.3. I agree with the reader that this evidence is circular. The coherence term in the Composite Score is cosine similarity between the refined topic vector and its associated word vectors; those same word vectors are neighbors of the topic in the hybrid graph and are averaged into the refined topic vector by the message-passing update in §4.5.2. The metric thus rewards the model for doing exactly what message passing does. The large coherence gain (English 0.470→0.613, French 0.404→0.496) is expected under the null hypothesis that the GNN merely averages neighbor embeddings. The rest of the Composite Score (Topic Diversity, Jaccard) is at best a diversity measure, not a semantic-quality measure, and no comparison to LDA/NMF/BERTopic on a shared metric is provided; the paper explicitly dismisses such comparison because outputs have different formats (§5). A second, smaller circularity is that the SGS weights are selected by silhouette score, a component of the later evaluation. There is no code or data artifact to re-run the pipeline. Credit is due for internally consistent methodology and readable, domain-appropriate topic examples in Appendix B.2, but those qualitative examples do not substitute for an independent quantitative anchor. Because the principal evidence of effectiveness is compromised and no external anchor replaces it, the REJECT verdict is appropriate; my stress-test does not change that verdict.","tokens_in":24152,"tokens_out":7981,"duration_ms":78766,"concrete_test":"Re-run the §5.3 ablation with all Word-in-Topic edges removed (or masked) during EdgeRefineGNN message passing, keeping document, similar-topic, similar-word edges, node features, and extraction fixed; then compute Coherence_t = avg(cosine_sim(emb_t, emb_associated_words)) and the Composite Score for Tables 17-24. If the refined-vs-original coherence gap (English 0.613 vs 0.470; French 0.496 vs 0.404) largely collapses, the claimed gain is an artifact of message passing over the words the metric scores against.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central evidence is the §5.3 ablation: EdgeRefineGNN-raised embeddings lift the Composite Score from 0.663±0.008 to 0.740±0.023 (English) and 0.633±0.029 to 0.743±0.018 (French), with the largest component gain in Coherence. But Coherence_t (§4.6) is avg(cosine_sim(emb_t, emb_associated_words)), where the 'associated words' are exactly the word nodes linked to t by Word-in-Topic edges (§4.4). In EdgeRefineGNN (§4.5.2), each topic update is h_i^{l+1}=σ(LayerNorm(1/|N(i)| Σ_{j∈N(i)} f_e(e_ij)h_j + h_i^l)): the refined topic embedding is a learned mean of its neighboring word, topic, and document embeddings. The metric therefore scores the model against the very vectors it was allowed to average in. The coherence gain is not independent evidence of semantic improvement. The paper explicitly forgoes a shared benchmark with LDA/NMF/BERTopic ('a direct comparison is irrelevant', §5), and no code or data are released, so nothing external validates the absolute Composite Score. Topic Diversity and low Jaccard overlap are diversity measures, not quality measures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HAMLET, a healthcare topic-modeling pipeline in which GPT-4o generates per-document topic phrases, SBERT-BERT hybrid embeddings represent documents and topics, a Semantic-Geometric Similarity (SGS) method builds similarity matrices, and an edge-conditioned GNN (EdgeRefineGNN) refines topic embeddings on a heterogeneous graph of documents, topics, and words. The final top-k topics are selected via k-means clustering followed by coherence-, centroid-, or connectivity-based extraction. The method is evaluated on three English and three French healthcare datasets of varying document lengths, using a newly introduced Composite Score, an ablation of refined versus original embeddings, and a statistical validation with t-tests and ANOVA. The paper claims that HAMLET produces interpretable, cross-lingual, document-length-insensitive topics without human intervention.","tokens_in":24412,"tokens_out":3049,"duration_ms":32100,"significance":"If the central effectiveness claim were supported, HAMLET would be a useful contribution to label-scarce healthcare text analysis, particularly because its outputs are short phrases usable as classification labels and because it is demonstrated on two languages. The paper also includes several positive features: six datasets with varying document lengths, repeated trials with descriptive statistics, t-tests, ANOVA, and sensitivity analysis of the Composite Score weights. However, no code or data are released, the evaluation metric is introduced in the same paper, and the main ablation evidence is confounded by the construction of the coherence metric. As a result, the paper currently does not provide independent evidence that HAMLET outperforms existing topic models or that EdgeRefineGNN improves semantic topic quality.","major_comments":[{"comment":"The central ablation claim is confounded by the coherence metric. Coherence_t is defined in §4.6 as avg(cosine_sim(emb_t, emb_associated_words)), where the associated words are exactly the word nodes connected to the topic by Word-in-Topic edges (§4.4). In EdgeRefineGNN's message passing (§4.5.2), the refined topic embedding is a learned mean-pooled aggregation over its neighboring word, topic, and document nodes, so the refinement directly moves topic embeddings toward the word embeddings used to score coherence. The reported gains in §5.3.1 (e.g., overall English coherence 0.470 for original versus 0.613 for refined, and Composite Score 0.663±0.008 versus 0.740±0.023) therefore do not constitute independent evidence of semantic improvement. The paper needs an external evaluation—standard coherence metrics on top words, human annotation, or downstream task performance—to support the claim that EdgeRefineGNN improves topic quality.","section":"§4.6, §4.7.3, §4.5.2, §5.3"},{"comment":"The central effectiveness claim is not tested against existing topic models on any common metric. The paper states that because HAMLET outputs phrases and existing models output bag-of-words topics, 'a direct comparison is irrelevant.' Yet Tables 5–10 report LDA, LSA, NMF, and BERTopic on Cv, Cnpmi, Topic Diversity, and Jaccard similarity, while HAMLET is evaluated only on the new Composite Score; no shared metric is used across both. The assertion that 'HAMLET already has an edge' is therefore unsupported. A head-to-head comparison on at least one standard metric, or a human evaluation of topic quality, is required before the claim of effectiveness can be accepted.","section":"§5 (intro) and §5.1–§5.2"},{"comment":"The Composite Score is introduced and weighted within the same paper, and the weights are selected from a sensitivity analysis on the same datasets used for the main evaluation. This makes the headline scores dependent on a criterion whose validity is not independently established. In particular, Topic Diversity receives weight 0.4, but diversity measures are not quality measures; high diversity can coexist with fragmented or meaningless topics. The paper should validate the Composite Score against established metrics or human judgments before using it to rank HAMLET's variants and to justify the ablation conclusion.","section":"§4.7.6, Tables 9–14"},{"comment":"The experimental setup is not reproducible as reported. No code, model weights, or processed data are released; the GPT-4o version, temperature, and other decoding parameters are not specified; the edge-feature encoding for the GNN is not fully defined; and the graph-construction thresholds are only partially described. Without these details, the statistical validation in §5.4 cannot be checked, and the reader's take that nothing external validates the absolute Composite Score remains correct. The authors should release code and data, and specify all experimental settings.","section":"§4.1–§4.7 and Appendices"}],"minor_comments":[{"comment":"The manuscript contains multiple typographical and grammatical errors, e.g., 'one key NLP task that enables that converts raw data' (§1), 'new approaches for for topic modeling' (§2.1), and 'an unbalanced and duplicates' in the reference list.","section":"Throughout"},{"comment":"The numbering is inconsistent: Figure 5 is captioned 'Figure 0', Table 3 is used twice (topics counts and graph network details), and the text references 'Table 171' for ANOVA results. The numbering should be corrected throughout.","section":"Table and figure numbering"},{"comment":"The IDF formula is garbled in the text: IDF(w)=log+Ncount(w)1 does not display the intended expression. The formula should be typeset correctly.","section":"§4.3.1"},{"comment":"The reference list duplicates entries (e.g., Blei et al. appears as [1] and [15]; Lee and Seung as [2] and [16]; several others are repeated), and a number of citations in the text are missing from the list. The reference list should be deduplicated and checked for completeness.","section":"References"},{"comment":"The Composite Score formula mixes terms with incompatible directions: Jaccard similarity is listed with a positive coefficient but the text states that lower values are better. The normalization direction should be stated explicitly for each component.","section":"§4.7.6"}],"recommendation":"reject","confidential_remarks":"The paper has a genuinely interesting architectural idea, but the evaluation is the manuscript's load-bearing component and it is not independent: the coherence metric is computed from the same graph edges that the GNN aggregates, and there is no shared benchmark against existing topic models. The authors explicitly dismiss direct comparison as 'irrelevant,' which leaves the central claim unvalidated. The issues could in principle be addressed with substantial new experiments (external metrics, human evaluation, code release), but that is beyond a revision of the current manuscript. I recommend rejection, with encouragement to resubmit after a fundamentally reworked evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a plausible pipeline but the main piece of evidence is circular. The stress-test note is right. The coherence component of the Composite Score is computed as the average cosine similarity between a topic embedding and the embeddings of its associated words. EdgeRefineGNN updates a topic embedding by mean-pooling messages from exactly those neighboring word nodes (plus other nodes). So after refinement, the topic embedding is literally closer to those words by construction. The 0.740 versus 0.663 ablation gain in Coherence is not independent evidence of semantic improvement; it's the model pulling the embeddings toward the thing it is then scored against.\n\nWhat's actually new: the specific combination of LLM-generated short phrases, SBERT-BERT hybrid embeddings, the SGS similarity (WMD-inspired plus cosine with IDF weighting), and an edge-conditioned GNN refinement. That combination is not in the cited prior work. The paper also runs a genuine cross-lingual study (English patient comments, French COVID news) across three document-length subsets each, and it includes a five-run statistical validation with t-tests and ANOVA. For a topic model meant to produce ready-to-use labels, getting interpretable short phrases is a real practical advantage. The approach is coherent and the experiments are described in enough detail to roughly reproduce, though they ship no code or data.\n\nSoft spots, in order of severity. First, the circularity above is load-bearing. The central claim that HAMLET is effective rests on the ablation, and the ablation's main component is confounded. Second, the paper explicitly declines a direct comparison with LDA/NMF/BERTopic, saying 'a direct comparison is irrelevant' (§5). That is a dodge. They could map HAMLET phrase topics to words and compute C_v or NPMI, or do a human evaluation. Without an external anchor, the absolute Composite Score is uninterpretable. Third, the SGS weights are tuned with silhouette score on the same topics that later contribute to a Composite Score that includes silhouette; the Composite weights themselves are chosen based on sensitivity analysis on the same six datasets. This is tuning-to-the-test-set, though its impact is unclear. Fourth, no code or data.\n\nI don't think this paper is ready for a serious referee. The architecture is worth a workshop paper or a preprint, and the authors show genuine thinking. But the evidence supporting the central claim is not independent. A revision needs a non-circular evaluation: compare against baselines on a shared metric, or report human-judged topic quality. If they do that, I'd be happy to look again. For now, I'd desk-reject rather than spend referee time.","headline":"HAMLET is a plausible LLM+GNN pipeline, but its main ablation gain is an artifact of scoring topic embeddings against the same graph neighbors the GNN averages in, so the central effectiveness claim is unsupported.","tokens_in":24957,"tokens_out":3007,"would_cite":false,"duration_ms":29511,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HAMLET uses GPT-4o-generated topics, then refines their embeddings with a graph neural network to deliver interpretable, cross-lingual healthcare topics that can serve as classification labels.","keywords":["topic modeling","large language models","graph neural networks","BERT","multilingual healthcare","embedding refinement","coherence"],"falsifier":"Train EdgeRefineGNN with an ablation that removes the word-in-topic edges from the graph (keeping only topic-topic, document-topic, and word-word edges), then compare the Composite Score gain over the original embeddings. If the gain mostly disappears when those edges are removed, the apparent coherence improvement is a direct consequence of the GNN's input graph, not of semantic refinement.","tokens_in":23912,"feed_emoji":"🔥","tokens_out":1817,"duration_ms":19332,"temperature":0.7,"pith_summary":"The paper tries to solve a real bottleneck: when healthcare text lacks labels, you need topics before you can classify, and traditional models like LDA or NMF produce overlapping, hard-to-interpret word bags. HAMLET proposes a pipeline where GPT-4o generates an initial pool of short topic phrases, a hybrid BERT–SBERT embedding scheme represents documents and topics, a graph neural network (EdgeRefineGNN) refines the topic embeddings by passing messages across document-topic-word edges, and a clustering plus coherence-based selection step extracts the top k topics. The authors claim this yields coherent, diverse, interpretable topics across English and French healthcare datasets, with refinement consistently improving a Composite Score over using the raw LLM embeddings. If correct, the approach would let practitioners skip manual topic naming and go straight to automated text classification in label-scarce healthcare settings.","feed_headline":"GPT-4o topics get a graph-neural polish for healthcare","feed_subtitle":"After the LLM proposes topics, a GNN refines their embeddings, yielding cross-lingual, label-ready topics on English and French corpora.","key_machinery":"The load-bearing mechanism is the EdgeRefineGNN, an edge-conditioned graph neural network that updates each topic node's embedding by mean-pooling messages from its neighboring word and topic nodes, where the message for each edge is produced by an MLP that takes the edge's type and similarity score as input. It is trained with an MSE loss against the original topic embeddings, so the refinement is a denoising step constrained to stay near the LLM's semantics while absorbing structural information from the graph. The graph itself is built from topic-assignment edges, similar-topic edges, similar-word edges, and word-in-topic edges, where similarities come from the proposed SGS method; this graph supplies the structural context the GNN uses to adjust embeddings.","core_discovery":"The paper's central claim is that topic embeddings produced by an LLM are raw and under-representative, and that refining them through a graph neural network that respects heterogeneous relationships—documents, topics, words, similar topics, and similar words—produces topics that are more coherent and better separated than the original embeddings. Evidence is offered through ablation: on the English datasets the refined embeddings with coherence-based extraction achieve an overall Composite Score of 0.740 ± 0.023 versus 0.663 ± 0.008 for the original embeddings, and the French overall scores are 0.743 ± 0.018 versus 0.633 ± 0.029. The paper also introduces a Semantic-Geometric Similarity (SGS) method that blends WMD-inspired word-level alignment with cosine similarity under IDF weighting, and a Composite Score metric that weights topic diversity, pairwise Jaccard similarity, coherence, silhouette, and Davies-Bouldin in order to evaluate phrase-form topics.","pith_inferences":["A cautious reading: the coherence component of the Composite Score, $\\text{Coherence}_t = \\text{avg}(\\cos(emb_t, emb_{words}))$, is computed after the GNN has already used word-in-topic edges in its message passing, so part of the observed coherence gain may be an artifact of the optimization target rather than independent semantic improvement. This is my inference as a reviewer, not a claim the p","A testable extension the authors do not propose: replace the coherence-based extraction with a holdout set of manually annotated topic labels (e.g., what a clinician would name the themes) and measure how often HAMLET's top-k phrases match those labels. That would directly validate the 'interpretable and usable as labels' claim.","If the SGS threshold (90th percentile of similarities) is truly adaptive, then the method should work across corpora with very different lexical densities; one could stress-test it by sampling a corpus with many near-duplicate articles (e.g., news wires) and checking whether the refined topics still collapse to a few redundant phrases.","The MSE loss that keeps refined embeddings close to the LLM's originals means the GNN cannot fully re-orient a topic that starts semantically wrong; a more aggressive refinement with an adversarial or contrastive loss might yield larger gains but also higher risk of drift. The authors do not explore this trade-off."],"forward_implications":["If the refinement claim holds, topic modeling pipelines that use LLM-generated phrases no longer need a human to interpret or name topics, since the extracted top-k phrases can be used directly as classification labels.","The approach is claimed to be language-agnostic in the tested sense, since the same pipeline with language-specific prompts worked on English patient feedback and French COVID news, producing stable Composite Scores across both.","The EdgeRefineGNN component, if genuinely beneficial, suggests that coupling an LLM's semantic priors with graph-structured message passing is a viable general recipe for unsupervised topic refinement, beyond the healthcare domain.","Since the ablation shows refined embeddings yield higher Silhouette and Davies-Bouldin scores, the refined embeddings may also serve better as features for downstream clustering or classification, not just as topic labels.","The SGS similarity method could be reused as a drop-in similarity measure for other phrase-level comparison tasks, since it combines word-level alignment, cosine similarity, and IDF weighting into a normalized score."],"supporting_citations":[{"why":"Supplies GPT-4o, the LLM used to generate the initial pool of topics in Step 1.","marker":"[11] OpenAI (2024)"},{"why":"Supplies BERT, whose token embeddings are used in the SBERT-BERT hybridization and for word embeddings.","marker":"[13] Devlin et al. (2018)"},{"why":"Supplies Sentence-BERT, the sentence-level embedding model combined with BERT in the hybridization.","marker":"[14] Reimers & Gurevych (2019)"},{"why":"Supplies the Word Mover's Distance idea that the SGS method adapts via cosine distance and IDF weighting.","marker":"[34] Kusner et al. (2015)"},{"why":"Supplies the Silhouette Score, one of the five metrics in the Composite Score and used for SGS weight optimization.","marker":"[35] Rousseeuw (1987)"},{"why":"Supplies the Davies-Bouldin Score, another component of the Composite Score.","marker":"[36] Davies & Bouldin (1979)"},{"why":"Provides the LDA baseline that HAMLET is compared against in the existing-models experiments.","marker":"[1] Blei et al. (2003)"},{"why":"Provides NMF, which turns out to be the strongest traditional baseline in the experiments, especially for French data.","marker":"[2] Lee & Seung (1999)"}],"fun_headline_variants":["LLM topics get GNN refinement for healthcare","GNN polishes LLM topic embeddings in healthcare","Cross-lingual healthcare topics via GNN-refined LLM","GNN refines LLM topics for coherent health models","Healthcare topics refined by GNN and LLM fusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's evidence that refinement improves quality rests on a Composite Score whose coherence component measures the average cosine similarity between a topic embedding and its associated word embeddings—and those exact 'word in topic' edges are already fed into the GNN during refinement, so the metric is not independent of the refinement mechanism it is used to validate.","fun_headline_variants_meta":{"raw":{"variants":["LLM topics get GNN refinement for healthcare","GNN polishes LLM topic embeddings in healthcare","Cross-lingual healthcare topics via GNN-refined LLM","GNN refines LLM topics for coherent health models","Healthcare topics refined by GNN and LLM fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1889,"prompt_tokens":1000,"completion_tokens":889,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":810}},"tokens_in":616,"tokens_out":889,"duration_ms":8595,"temperature":1.0,"reasoning_tokens":810,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:23:12.870080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train EdgeRefineGNN with an ablation that removes the word-in-topic edges from the graph (keeping only topic-topic, document-topic, and word-word edges), then compare the Composite Score gain over the original embeddings. If the gain mostly disappears when those edges are removed, the apparent coherence improvement is a direct consequence of the GNN's input graph, not of semantic refinement.","supporting_citations":[],"review_version":1}