Pith. sign in

REVIEW 3 major objections 5 minor 19 references

HERCULES: Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Recursive k-means plus LLM-written labels turns abstract clusters into a readable tree of topics.

desk verdict A transparent, well-engineered clustering tool that clearly credits its borrowed core, but the interpretability payoff is asserted rather than measured. read the letter →

arxiv 2506.19992 v2 pith:CU6IK7HP submitted 2025-06-24 cs.LG cs.AI

Gabor Petnehazi , Bernadett Aradi This is my paper
classification cs.LGcs.AI
keywords hierarchicalclusteringk-meanslargelanguagemodelsclustersummarizationembeddinginterpretabletopic-guidedtext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hercules claims that a classic recursive k-means hierarchy becomes genuinely interpretable when every cluster at every level is given a short LLM-generated title and description. The paper argues that the algorithm's value is not a new clustering geometry but a bridge from abstract vector groupings to human narratives, built from representative samples, child-cluster summaries, and numeric statistics. On the 20 Newsgroups benchmark the method is competitive with flat embedding clustering while adding a multi-level, topic-labeled tree. The authors also test a 'description' mode in which higher levels cluster the embeddings of the previous level's summaries rather than the original data. A sympathetic reading is that the system's core contribution is an LLM-in-the-loop summarization pipeline wrapped around standard k-means.

What carries the argument

The machinery is a recursive k-means loop in which the representation of a parent cluster is the centroid of its children, with an optional iterative resampling step that re-runs k-means on core members to refine centroids. After each level, an adaptive LLM prompt is assembled per cluster from representative level-0 items, sampled immediate child summaries, optional numeric statistics, and an optional 'topic seed' that orients the labeling; responses are parsed as JSON and embedded for the next level when description mode is active. This prompt pipeline is what carries the interpretability claim, because every node of the tree ends up carrying a short human-readable label grounded in the data below it.

What would settle it

Ask human evaluators to inspect a random sample of clusters from the 20 Newsgroups runs and mark each LLM title as accurate, partial, or wrong against the cluster's actual documents; if a substantial share of titles are marked wrong or partial, the interpretability claim fails. A cheaper version: hold out half of each cluster's members, generate the title from the other half, and measure how often the title matches the held-out members better than titles from other clusters.

Watch

Extended reading notes

Core claim

The paper's central discovery is that LLM-generated summaries can be made the organizing interface of a hierarchical cluster tree. Starting from individual items at level 0, Hercules applies k-means recursively, and after each merge it prompts an LLM with representative level-0 samples, immediate children's titles and descriptions, and (for numeric data) statistics, asking for a concise title and one-to-two-sentence description per cluster. In 'direct' mode clustering uses the original item embeddings; in 'description' mode higher levels cluster the embeddings of the LLM summaries themselves, letting the model's semantic reading steer the hierarchy. The authors report that on the 20 Newsgroups corpus the resulting three-level tree yields top-level themes such as hardware, sports, and religion, with the LLM also isolating an 'empty posts' cluster, and that the best Hercules configuration reaches an adjusted Rand index of 0.405 against 0.468 for flat k-means on the same embeddings. They read this as competitive performance plus an interpretability bonus that the metrics do not capture.

Load-bearing premise

The load-bearing premise is that a few sampled items, plus the previous level's summaries, give the LLM enough information to write a title and description that faithfully represent the whole cluster; the paper does not verify this with human judges.

Editorial extensions

If this is right

  • A user can explore a large corpus top-down: broad themes at the top, finer sub-themes at each lower level, without reading the raw items.
  • Because every cluster carries a description embedding, cluster quality can be assessed in summary space, not only in original feature space.
  • The topic seed gives analysts a way to steer labels toward their research question, and in description mode this steering propagates up the hierarchy.
  • The same code path handles text, images, and numeric vectors one modality at a time, so the interpretability layer transfers across data types.
  • If the summaries are trustworthy, the hierarchy itself becomes a deliverable: a structured, human-readable map of an unlabeled dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The interpretability claim is only as strong as the summary fidelity; a direct extension would be a human evaluation where annotators judge whether each title covers all or almost all members of its cluster.
  • Description mode's lower ARI in the paper may understate its value: the summaries could be judged on whether they capture the data's actual themes, not on recovering the original newsgroup labels.
  • The pipeline could be applied to any recursive partition, not just k-means; replacing k-means with another hierarchical algorithm would preserve the LLM-summarization contribution.
  • The interactive visualization suggests a testable workflow: users who navigate by LLM titles should find relevant documents faster than users navigating raw embeddings or PCA coordinates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces HERCULES, a hierarchical k-means clustering algorithm that recursively partitions data (text, images, or numeric vectors, one modality per run) and uses LLMs to generate a title and description for every cluster at every level. The method supports a 'direct' mode (clustering on original embeddings or scaled features) and a 'description' mode (clustering on embeddings of LLM-generated summaries), an optional topic seed to guide summarization, iterative centroid resampling, and a companion Python package with evaluation metrics and an interactive Dash visualization app. The authors evaluate the method on the 20 Newsgroups dataset, comparing a representative HERCULES configuration with LSA-based and flat embedding-based baselines, and report a 2^5 factorial ablation over representation mode, embedding model, LLM model, resampling, and topic seed. The central claim is that LLM-generated summaries at each hierarchy level significantly enhance interpretability compared with flat clustering.

Significance. The idea of combining recursive k-means with LLM-generated cluster explanations is timely and the open-source implementation (with visualization and evaluation tooling) is a potentially useful practical contribution. The paper is clearly written and the algorithmic details are transparent. However, the main quantitative evidence does not yet substantiate the central claim: HERCULES is slightly worse than flat CloudEmbeddings+KMeans on ARI/NMI, and the proposed interpretability advantage is supported only by anecdotal examples and by self-referential embedding-based metrics that share the same embedding model used for clustering. If a rigorous human evaluation (or an equivalent fidelity test against ground-truth topic descriptions) were added and showed genuine improvements in human understanding, the paper would be a solid systems contribution. As it stands, the headline claim is plausible but unverified.

major comments (3)
  1. [§7.2, §7.3] The paper's central claim—stated in the abstract and reiterated in §8—is that LLM-generated titles and descriptions 'significantly enhance interpretability.' This claim is supported only by the qualitative examples in §7.3 from a single dataset and by the LLM Silhouette score in §7.2. No human evaluation, structured user study, or comparison of generated summaries against ground-truth topic descriptions is provided. The LLM Silhouette is computed on embeddings of the summaries themselves and therefore measures distinctiveness of the summaries in an embedding space, not their fidelity or usefulness to a human reader. A user study with ratings of title/description accuracy, informativeness, or usefulness for identifying cluster content is required to support the headline claim.
  2. [§7.1] The head-to-head comparison in Table 1 shows that HERCULES (direct mode, cloud models) achieves ARI 0.405 and NMI 0.591, whereas flat CloudEmbeddings+KMeans achieves ARI 0.468 and NMI 0.618. The text acknowledges this but argues that HERCULES provides a 'full interpretable hierarchy' as a qualitative advantage. Since interpretability is never measured, the empirical record currently shows only a modest accuracy loss without any demonstrated benefit. The comparison would be persuasive if interpretability were quantified on the same runs, e.g., by having users identify cluster content or verify titles against held-out items.
  3. [§3.4, §3.7] The fidelity of the LLM summaries depends on the representative L0 samples and child summaries included in the prompt, yet no sensitivity analysis is reported for prompt_l0_sample_size, sampling strategy, or prompt_immediate_child_sample_size. For large clusters (such as the 'Replies, Responses, and General Discussions' cluster in §7.3), the sampled items may not reflect the cluster's overall content, and higher-level summaries built from child summaries may compound sampling errors. Additionally, the Topic Alignment Score defined in §3.7 uses the same embedding model family that produced the cluster representations and description embeddings; it measures geometric alignment in that embedding space, not semantic fidelity to a human topic. A sensitivity analysis, ideally combined with human-rated summary fidelity, is needed to establish that the generated summaries are not artifacts of an unrepresentative sample.
minor comments (5)
  1. [§4, Algorithm 3] In Algorithm 3, the variable 'L' is first used as the LLM client in Algorithm 1 and in line 31, but line 12 assigns the k-means output to 'L, Ccentroids', shadowing the LLM client with a labels variable. Please rename the labels variable (e.g., 'lab') to avoid confusion.
  2. [§6.3, Table 1] The reported 'Clustering Time' is difficult to interpret because the text notes that HERCULES runtimes benefited from an embedding cache, but it is not clear whether the reported values include LLM summarization time or only the k-means loop. Please clarify what the timing includes.
  3. [Throughout] The paper uses both 'HERCULES' (abstract, title) and 'Hercules' (body text) inconsistently; please standardize the capitalization.
  4. [§7.1] The phrase 'dramatic improvement of more than 7x over the best LSA baseline' is misleading because the embedding baselines also show large improvements over LSA; the comparison is really about embedding-based methods versus LSA, not about HERCULES specifically.
  5. [§7.2] For description mode, the LLM Silhouette is computed on the same description embeddings used for clustering; this should be acknowledged as a coherence measure for the summarization process rather than a measure of interpretation quality.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: HERCULES's recursive k-means core is externally credited, and its quantitative claims rest on standard external ARI/NMI comparisons rather than fitted parameters.

full rationale

The paper's central chain is algorithmic rather than derived: HERCULES applies recursive k-means (explicitly attributed to Vo et al. 2024) and augments each cluster with LLM-generated titles/descriptions. No parameter is fitted to the evaluation targets, no self-citation provides load-bearing justification, and no uniqueness theorem is imported from the authors' prior work. The main quantitative evidence against baselines uses standard external metrics (ARI, NMI) on 20 Newsgroups ground-truth labels, which are independent of the method's internals. The LLM-silhouette and topic-alignment scores are embedding-based proxies and could be questioned on validity grounds (e.g., the same embedding family is used for clustering and for evaluating summary separability, and interpretability is not measured by human raters), but those are correctness/validation limitations, not circular reductions by construction. The derivation chain is therefore self-contained.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The algorithm itself does not introduce new physical or mathematical entities. The free parameters are all configuration choices for the clustering and prompting, none of which are fitted to external data. The axioms are standard assumptions about k-means, embeddings, and LLM behavior that the method depends on without independent verification.

free parameters (4)
  • level_cluster_counts = [100, 20, 5]
    Hand-chosen number of clusters at each hierarchy level; determines granularity and allows comparison to the 20 ground-truth classes. The paper sets this explicitly in Section 6.3.
  • use_resampling = True/False
    Boolean toggle for centroid refinement; the ablation shows it affects LLM silhouette scores (Section 7.2).
  • representation_mode = 'direct' or 'description'
    Choice of which vectors to cluster; directly affects ARI (0.303 vs 0.172 in Table 2).
  • prompt_l0_sample_size = Not specified
    Number of representative items shown to the LLM; a configurable parameter that affects the summary quality.
assumptions (4)
  • standard math K-means minimizes within-cluster variance on the given representations.
    Used in Algorithm 3 without proof; standard result.
  • domain assumption Pre-trained embedding models produce vectors whose similarities reflect semantic similarity.
    The whole method relies on this to group texts and images; stated in Sections 3.2 and 6.3.
  • domain assumption LLMs return valid JSON and produce titles and descriptions that are faithful summaries of the provided context.
    Required for the summarization module (Section 3.4); not rigorously validated.
  • domain assumption The 20 Newsgroups dataset labels correspond to sensible clusters.
    Used as ground truth for external metrics (Section 6.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of HERCULES: Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization." pith.science (2026). https://pith.science/paper/CU6IK7HP

@misc{pith2026250619992,
  author       = {Pith},
  title        = {Pith review of: HERCULES: Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CU6IK7HP}},
  note         = {Machine review of arXiv:2506.19992}
}
read the original abstract

The explosive growth of complex datasets across various modalities necessitates advanced analytical tools that not only group data effectively but also provide human-understandable insights into the discovered structures. We introduce HERCULES (Hierarchical Embedding-based Recursive Clustering Using LLMs for Efficient Summarization), a novel algorithm and Python package designed for hierarchical k-means clustering of diverse data types, including text, images, and numeric data (processed one modality per run). HERCULES constructs a cluster hierarchy by recursively applying k-means clustering, starting from individual data points at level 0. A key innovation is its deep integration of Large Language Models (LLMs) to generate semantically rich titles and descriptions for clusters at each level of the hierarchy, significantly enhancing interpretability. The algorithm supports two main representation modes: `direct' mode, which clusters based on original data embeddings or scaled numeric features, and `description' mode, which clusters based on embeddings derived from LLM-generated summaries. Users can provide a `topic\_seed' to guide LLM-generated summaries towards specific themes. An interactive visualization tool facilitates thorough analysis and understanding of the clustering results. We demonstrate HERCULES's capabilities and discuss its potential for extracting meaningful, hierarchical knowledge from complex datasets.

Figures

Figures reproduced from arXiv: 2506.19992 by the authors.

Figure 1
Figure 1. The five top-level clusters (Level 3) discovered by [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. The 20 clusters at Level 2. Clusters are colored by their Level 3 parent, showing [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. The 100 most granular clusters at Level 1. This view reveals the fine-grained [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages

  1. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information pro- cessing systems, 33:1877–1901, 2020

    work page 1901

  2. [2]

    Bert: Pre- training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceed- ings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    work page 2019

  3. [3]

    Text clustering as classification with llms

    Chen Huang and Guoxiu He. Text clustering as classification with llms. arXiv preprint arXiv:2410.00927, 2024

    arXiv 2024

  4. [4]

    Data clustering: a review

    Anil K Jain, M Narasimha Murty, and Patrick J Flynn. Data clustering: a review. ACM computing surveys (CSUR) , 31(3):264–323, 1999

    work page 1999

  5. [5]

    Image clustering conditioned on text criteria

    Sehyun Kwon, Jaeseung Park, Minkyu Kim, Jaewoong Cho, Ernest K Ryu, and Kangwook Lee. Image clustering conditioned on text criteria. arXiv preprint arXiv:2310.18297, 2023

    arXiv 2023

  6. [6]

    Concept induction: Analyzing unstructured text with high-level concepts using lloom

    Michelle S Lam, Janice Teoh, James A Landay, Jeffrey Heer, and Michael S Bern- stein. Concept induction: Analyzing unstructured text with high-level concepts using lloom. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–28, 2024

    work page 2024

  7. [7]

    Human-interpretable clustering of short text using large language models

    Justin K Miller and Tristram J Alexander. Human-interpretable clustering of short text using large language models. Royal Society Open Science , 12(1):241692, 2025

    work page 2025

  8. [8]

    Twenty Newsgroups

    Tom Mitchell. Twenty Newsgroups. UCI Machine Learning Repository, 1997. DOI: https://doi.org/10.24432/C5C323

    doi:10.24432/c5c323 1997

Show all 19 references
  1. [9]

    A data and analytics web app framework for python, no javascript required

    Chris Parmer, Philippe Duval, and Alex Johnson. A data and analytics web app framework for python, no javascript required

  2. [10]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blon- del, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Courna- peau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Lear...

    2011

  3. [11]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Interna- tional conference on machine learning...

    2021

  4. [12]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 , 2019. 21

    1908 arXiv

  5. [13]

    Clio: Privacy-preserving insights into real-world ai use

    Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, et al. Clio: Privacy-preserving insights into real-world ai use. arXiv preprint arXiv:2412.13678 , 2024

    2024 arXiv

  6. [14]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  7. [15]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´ e, Morgane Rivi` ere, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 , 2025

    2025 arXiv

  8. [16]

    Context-aware clustering using large language models

    Sindhu Tipirneni, Ravinarayana Adkathimar, Nurendra Choudhary, Gaurush Hi- ranandani, Rana Ali Amjad, Vassilis N Ioannidis, Changhe Yuan, and Chandan K Reddy. Context-aware clustering using large language models. arXiv preprint arXiv:2405.00988, 2024

    2024 arXiv

  9. [17]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  10. [18]

    Large language models enable few-shot clustering

    Vijay Viswanathan, Kiril Gashteovski, Kiril Gashteovski, Carolin Lawrence, Tong- shuang Wu, and Graham Neubig. Large language models enable few-shot clustering. Transactions of the Association for Computational Linguistics , 12:321–333, 2024

    2024

  11. [19]

    Automatic data curation for self-supervised learning: A clustering-based ap- proach

    Huy V Vo, Vasil Khalidov, Timoth´ ee Darcet, Th´ eo Moutakanni, Nikita Smetanin, Marc Szafraniec, Hugo Touvron, Camille Couprie, Maxime Oquab, Armand Joulin, et al. Automatic data curation for self-supervised learning: A clustering-based ap- proach. arXiv preprint arXiv:2405.1...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.