{"arxiv_id":"2611.17030","title":"Similar but Not Redundant: Embedding-Gated Refinement for High-Granularity Topic Models","abstract":"High-granularity topic modelling extracts hundreds of niche topics from a corpus, yet this regime is rarely studied and is more commonly framed as a failure mode: \"over-clustering,\" in which coherent themes fragment into redundant subtopics. We argue that this framing conflates genuine redundancy with topics that are semantically similar but meaningfully distinct, and that the central challenge in high-granularity modelling is separating the two. Recent post-hoc refinement methods address topic redundancy through either embedding-based similarity thresholding or LLM-guided merging. We show that neither suffices in high-granularity settings and propose embedding-gated iterative merge (EGIM), a hybrid approach that iteratively generates context-rich topic representations, uses embedding similarity to propose candidate merges across the topic space, then applies LLM judgement to accept only genuinely redundant pairs while preserving niche distinctions. Across two patent corpora and one legal corpus, LLM judgement rejects 83% of embedding-identified candidates as non-redundant. EGIM improves discriminability on merged topics by 25-55% over a prior iterative LLM baseline while better preserving the original topic distribution.","primary_cat":"cs.CL","cross_cats":["cs.IR"],"submitted_at":"2026-11-12T16:17:08+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"CONDITIONAL","confidence":"MODERATE","one_line_summary":"EGIM, a hybrid post-hoc refinement method, uses embedding-similarity gating plus LLM merge/keep judgments to remove redundant topics while preserving semantically similar niche topics.","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0,"inbound_count":0,"_links":{"html":"https://pith.science/paper/2611.17030","verdict":"/api/v1/verdicts/2611.17030"}}