pith. sign in

arxiv: 2604.14489 · v2 · submitted 2026-04-15 · 💻 cs.CL

CobwebTM: Probabilistic Concept Formation for Lifelong and Hierarchical Topic Modeling

Karthik Singaravadivelan , Anant Gupta , Zekun Wang , Christopher J. MacLellan This is my paper

Pith reviewed 2026-05-10 12:38 UTC · model grok-4.3

classification 💻 cs.CL
keywords topic modelinglifelong learninghierarchical topic modelsconcept formationincremental learningprobabilistic modelsdocument embeddingsCobweb algorithm
0
0 comments X

The pith

Adapting the Cobweb algorithm to document embeddings creates a low-parameter lifelong hierarchical topic model that discovers topics dynamically.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Topic modeling aims to find hidden semantic structures in text with little supervision. Traditional neural methods perform well but demand much tuning and suffer from forgetting in lifelong settings, while older probabilistic models cannot adapt easily to new data streams. This paper presents CobwebTM, which takes the classic Cobweb concept formation algorithm and modifies it to work directly with pretrained document embeddings. The result is an online system that builds topic hierarchies incrementally, creates new topics as needed, and keeps topics stable without needing a preset number of topics. Experiments on various datasets show competitive coherence and hierarchy quality, suggesting this hybrid symbolic-neural approach offers an efficient alternative for lifelong topic modeling.

Core claim

CobwebTM is a lifelong hierarchical topic model based on incremental probabilistic concept formation adapted to continuous document embeddings. It constructs semantic hierarchies online without predefining the number of topics, supports dynamic topic creation, and maintains stability over time while achieving strong topic coherence.

What carries the argument

Incremental probabilistic concept formation from the Cobweb algorithm, applied to continuous embeddings by mapping them into discrete probabilistic splits.

If this is right

  • Strong topic coherence is achieved across diverse datasets
  • Topics remain stable over time in streaming scenarios
  • High-quality hierarchies are produced without predefined topic counts
  • The model operates with low parameters and minimal tuning
  • Unsupervised topic discovery and dynamic creation are enabled in lifelong settings

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This approach might extend to other representation types beyond pretrained embeddings
  • It could reduce computational costs compared to retraining neural models for new data
  • Connections to human-like incremental learning in cognitive science could be explored
  • Integration with modern embedding models might further improve performance

Load-bearing premise

The mapping from continuous document embeddings to the discrete probabilistic category splits in the original Cobweb algorithm preserves coherence and stability without introducing instabilities or demanding heavy hyperparameter adjustments.

What would settle it

Observing significant degradation in topic coherence or sudden topic instability when processing a long stream of new documents without retuning would falsify the central claim.

Figures

Figures reproduced from arXiv: 2604.14489 by Anant Gupta, Christopher J. MacLellan, Karthik Singaravadivelan, Zekun Wang.

Figure 1
Figure 1. Figure 1: A visualization of three levels of the hierarchy induced by [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cv comparison on the StackOverflow, Spatiotemporal News, and TweetNER dataset. Baselines. We compare against three primary baselines: TraCo (Wu et al., 2024d), a neural model using Optimal Transport for topic regular￾ization; BoxTM (Lu et al., 2024), a geometric ap￾proach modeling topics as hyper-rectangles, and BERTopic (Hierarchical) (Grootendorst, 2022), which uses agglomerative clustering on top of fla… view at source ↗
Figure 3
Figure 3. Figure 3: ARI comparison on the StackOverflow, Spatiotemporal News, and TweetNER dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: TCD comparison on the StackOverflow, Spatiotemporal News, and TweetNER dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Intruder Similarity (ISIM) comparison on the StackOverflow, Spatiotemporal News, and TweetNER [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An ablation study to compare the lifelong results of using different embedding models for [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An ablation study to compare the lifelong results of [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: An ablation study to compare the amount of each Cobweb operation per-batch for all three datasets in [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
read the original abstract

Topic modeling seeks to uncover latent semantic structure in text corpora with minimal supervision. Neural approaches achieve strong performance but require extensive tuning and struggle with lifelong learning due to catastrophic forgetting and fixed capacity, while classical probabilistic models lack flexibility and adaptability to streaming data. We introduce CobwebTM, a low-parameter lifelong hierarchical topic model based on incremental probabilistic concept formation. By adapting the Cobweb algorithm to continuous document embeddings, CobwebTM constructs semantic hierarchies online, enabling unsupervised topic discovery, dynamic topic creation, and hierarchical organization without predefining the number of topics. Across diverse datasets, CobwebTM achieves strong topic coherence, stable topics over time, and high-quality hierarchies, demonstrating that incremental symbolic concept formation combined with pretrained representations is an efficient approach to topic modeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces CobwebTM, a lifelong hierarchical topic model obtained by adapting the Cobweb incremental probabilistic concept-formation algorithm to continuous document embeddings produced by pretrained language models. It claims that the resulting system performs unsupervised topic discovery, creates topics dynamically, organizes them hierarchically, and does so with low parameter count and without pre-specifying the number of topics, while delivering strong coherence, temporal stability, and high-quality hierarchies on diverse datasets.

Significance. If the central empirical claims are substantiated, the work would be significant for lifelong learning in NLP: it supplies a concrete, incremental symbolic mechanism that sidesteps catastrophic forgetting and fixed-capacity issues of neural topic models while retaining the representational power of pretrained embeddings. The approach is distinctive in its use of an established concept-formation algorithm rather than purely neural or Bayesian nonparametric alternatives.

major comments (2)
  1. [§3.2] §3.2 (Probabilistic splits on continuous embeddings): the mapping from continuous document vectors to Cobweb-style attribute-value probabilities is described only at a high level; no explicit formula, kernel, or distance threshold is given, nor is it shown to be parameter-free. Because this mapping is load-bearing for the stability and low-parameter claims, its definition must be stated precisely (ideally with a derivation or pseudocode) so that readers can verify it does not introduce hidden hyperparameters or dataset-specific tuning.
  2. [§4] §4 (Experiments): the abstract asserts 'strong topic coherence, stable topics over time, and high-quality hierarchies,' yet the reported results lack (i) direct comparison against strong lifelong baselines (e.g., dynamic topic models or online neural topic models), (ii) ablation on the choice of embedding model, and (iii) quantitative measures of hierarchy quality (e.g., dendrogram purity or topic hierarchy coherence). These omissions make it impossible to evaluate whether the claimed advantages are realized or merely asserted.
minor comments (2)
  1. [Abstract] The abstract would be strengthened by including one or two key quantitative results (coherence scores, stability metrics) rather than qualitative adjectives.
  2. [§3] Notation for the adapted Cobweb probability update (Eq. X) should be aligned with the original Cobweb paper to facilitate comparison.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. The comments highlight important areas for improving technical clarity and experimental rigor. We address each major comment point by point below and have revised the manuscript to incorporate the suggested changes.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (Probabilistic splits on continuous embeddings): the mapping from continuous document vectors to Cobweb-style attribute-value probabilities is described only at a high level; no explicit formula, kernel, or distance threshold is given, nor is it shown to be parameter-free. Because this mapping is load-bearing for the stability and low-parameter claims, its definition must be stated precisely (ideally with a derivation or pseudocode) so that readers can verify it does not introduce hidden hyperparameters or dataset-specific tuning.

    Authors: We agree that the description of the mapping in §3.2 is high-level and requires greater precision to support the stability and low-parameter claims. In the revised manuscript, we will include an explicit mathematical formula for converting continuous document embeddings into attribute-value probabilities, specify the kernel or distance threshold employed, and provide pseudocode for the probabilistic split mechanism. We will also add a short derivation demonstrating that the mapping introduces no new hyperparameters or dataset-specific tuning, relying solely on properties of the pretrained embeddings. This will enable readers to verify the parameter-free nature of the approach. revision: yes

  2. Referee: [§4] §4 (Experiments): the abstract asserts 'strong topic coherence, stable topics over time, and high-quality hierarchies,' yet the reported results lack (i) direct comparison against strong lifelong baselines (e.g., dynamic topic models or online neural topic models), (ii) ablation on the choice of embedding model, and (iii) quantitative measures of hierarchy quality (e.g., dendrogram purity or topic hierarchy coherence). These omissions make it impossible to evaluate whether the claimed advantages are realized or merely asserted.

    Authors: We acknowledge that the experimental section would benefit from additional comparisons and quantitative analyses to more fully substantiate the claims. The current results already include coherence and stability metrics along with qualitative hierarchy evaluations across multiple datasets, but we agree that direct comparisons to strong lifelong baselines such as dynamic topic models and online neural topic models are needed. In the revision, we will add these comparisons, include an ablation study varying the pretrained embedding model, and report quantitative hierarchy quality measures including dendrogram purity and topic hierarchy coherence. These additions will provide stronger empirical grounding for the advantages of CobwebTM in lifelong and hierarchical settings. revision: yes

Circularity Check

0 steps flagged

No circularity: algorithmic adaptation of independent Cobweb framework

full rationale

The paper describes CobwebTM as an incremental adaptation of the pre-existing Cobweb algorithm (Fisher 1987) to continuous document embeddings from pretrained models. No equations, derivations, or first-principles results are presented that reduce any claimed prediction or hierarchy property to fitted parameters or self-referential definitions by construction. The core claims rest on the original Cobweb's probabilistic splits (independent prior work) plus external pretrained representations, with no load-bearing self-citations or ansatz smuggling. The approach is presented as an engineering combination rather than a closed mathematical derivation, making it self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Only the abstract is available, so the ledger is limited to assumptions stated or implied there. No free parameters or invented entities are explicitly named.

axioms (1)
  • domain assumption Pretrained document embeddings capture sufficient semantic similarity to support probabilistic concept splits originally designed for symbolic features.
    The adaptation of Cobweb to continuous embeddings rests on this premise.

pith-pipeline@v0.9.0 · 5434 in / 1158 out tokens · 19232 ms · 2026-05-10T12:38:22.174093+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou

    Topic modeling in embedding spaces.Trans- actions of the Association for Computational Linguis- tics, 8:439–453. Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. 2021. Sawtooth factorial topic embeddings guided gamma belief network.CoRR, abs/2107.02757. Douglas H. Fisher. 1987. Knowledge acquisition v...

    work page arXiv 2021

  2. [2]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Efficient and scalable masked word predic- tion using concept formation.Cognitive Systems Research, 92:101371. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.Preprint, arXiv:1907.11692. Yuyin Lu, Hegang...

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  3. [3]

    INSERT" operation, necessary for traversing the tree as a whole. Notably, the

    Hyhtm: Hyperbolic geometry based hierar- chical topic models.Preprint, arXiv:2305.09258. Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco. Barbieri, and Jose Camacho-Collados. 2022. Named Entity Recognition in Twitter: A Dataset and Anal- ysis on Short-Term Temporal Shifts. InThe 2nd Conference of the Asia-Pacific Chapter of the Asso- ciation for Compu...

    work page arXiv 2022