Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper maps which implementation choices matter for the Dynamic Embedded Topic Model (DETM) and finds that recomputing window statistics and reweighting the loss have no significant or consistent effect on test-set negative…

desk verdict Useful empirical map of DETM hyperparameter sensitivity, but the abstract overstates the null result on loss reweighting, and the significance threshold is carried across corpora on thin ice. read the letter →

arxiv 2504.19209 v1 pith:OGS6DQZ7 submitted 2025-04-27 cs.CL cs.LG

Elisabeth Fittschen , Bella Xia , Leib Celnik , Paul Dilley , Tom Lippincott This is my paper · ORCID
classification cs.CLcs.LG
keywords dynamicembeddedtopicmodeldiachroniccorporanegativelog-likelihoodhyperparametersensitivityvocabularyscalingtemporalwindowsmodelssemanticchange
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks which implementation choices actually matter when applying the Dynamic Embedded Topic Model (DETM) to historical text. Using five diachronic corpora covering ancient Greek, Latin, early science fiction, ACL papers, and UN proceedings, it measures test-set negative log-likelihood under controlled sweeps of five choices: recomputing window statistics, reweighting the loss, delta ratios, window count, and vocabulary size. The central finding is that two choices that complicate deployment—recomputing time-window statistics and reweighting the loss with a global batch ratio—do not significantly or consistently affect performance, so they can be dropped. Vocabulary size can be scaled from 5,000 to 80,000 with almost no change in NLL, and smaller window counts perform best, though the model degrades gracefully with more windows. The paper positions these results as an initial map for applied scholars and for a future continuous-time variant.

What carries the argument

The central object is the Dynamic Embedded Topic Model (DETM), a neural topic model that combines word embeddings with a dynamic topic model: each time window has a topic-mixture prior and topic embeddings that evolve by random walks, and a variational inference network maps sub-documents to topic mixtures. The paper's instrument is controlled before-and-after comparison of test-set per-word negative log-likelihood, with a Bessel-corrected estimate of $2\sigma = 0.03$ derived from 20 seeded ACL runs to set a significance threshold. The argument works by holding all other hyperparameters at their defaults and sweeping one factor per corpus, then checking whether differences clear the threshold.

What would settle it

Run 20 seeded replicates of the same configuration on each of greek, latin, scifi, and un, and compute the corpus-specific 2σ interval for test NLL. If that interval is wider than the observed differences—for instance, if latin's roughly 0.21 reweighting gap falls inside seed noise—then the paper's conclusion that these choices have no significant or consistent effect would not hold for those corpora.

Watch

Extended reading notes

Core claim

The paper establishes that DETM's test-set negative log-likelihood is not significantly or consistently affected by recomputing the mixture-prior summary statistics at validation or test time, nor by reweighting the reconstruction and KL terms by the ratio of training documents to batch size. Across five corpora, NLL differences from these choices fall near or below a $2\sigma = 0.03$ threshold estimated from 20 seeded ACL runs, and where differences are larger they do not favor one option consistently. The model is stable when vocabulary size grows from 5,000 to 80,000 words, continues to improve with larger topic inventories up to 80–160 topics on several corpora, and performs best with 2–4 temporal windows while remaining robust to over-granular windows through an interpolation policy for empty windows. $\Delta$ ratios show no interpretable pattern except that the highest ratio is consistently worst.

Load-bearing premise

The significance threshold used to decide whether differences matter is estimated from 20 seeded runs on the ACL corpus only and then applied to the other four corpora, each of which is tested with a single run per configuration.

Editorial extensions

If this is right

  • Practitioners can turn off recomputation of window statistics and loss reweighting, eliminating the need to track global batch statistics during training and inference.
  • DETM's predictive performance is essentially flat as vocabulary size grows from 5,000 to 80,000, so the practical limit is memory in constructing normalized categorical topic distributions rather than model quality.
  • Fewer temporal windows (2 or 4) give the best test NLL, and the interpolation policy keeps over-granular windows from causing catastrophic drift, supporting the pursuit of a continuous-time variant.
  • Larger topic inventories continue to improve NLL on several corpora, with no single optimum across all five; some corpora benefit well beyond 80 topics.
  • Delta-ratio choices have no consistent interpretation except that a 9:1 ratio is consistently worst, so defaulting to equal deltas is a reasonable starting point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper measures NLL rather than human-rated topic quality, its recommendation to drop recomputation and reweighting concerns predictive fit; a practitioner whose goal is interpretable topics should still check that those changes do not alter the topics surfaced.
  • The stability across vocabulary sizes suggests DETM is a good fit for historical text with orthographic variation and pre-standardized spelling, provided the softmax memory bottleneck is addressed; the paper notes this direction but does not test it.
  • A cheap protocol for future applied work follows from the significance caveat: before comparing configurations on a new corpus, run a small set of seeded replicates to calibrate corpus-specific noise rather than reusing the ACL threshold.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents an empirical study of implementation choices for Dynamic Embedded Topic Models (DETM) across five diachronic corpora (ACL, ancient Greek, Latin, science fiction, and UN). The authors vary six factors: recomputation of time-window statistics, loss reweighting, topic count, window count, vocabulary size, and the ratio of mixture-walk to topic-walk deltas, measuring held-out test-set negative log-likelihood (NLL), with NPMI coherence reported in an appendix. They conclude that several implementation aspects do not significantly or consistently affect NLL, that DETM is stable under vocabulary scaling, that smaller window counts perform best, and that larger topic counts generally improve performance. The paper releases standardized datasets, experimental code, and a pip-installable library.

Significance. The paper's main strength is its multi-corpus, held-out evaluation of practical DETM choices, addressing questions that arise in applied and humanistic scholarship. The release of datasets, code, and a library is a valuable contribution to reproducibility. If the empirical claims are fully supported after revision, the paper would be a useful map for practitioners. However, the inferential basis for the 'not significant' language is currently limited, and the abstract overstates the reweighting results. I therefore view the contribution as promising but requiring substantial qualification.

major comments (3)
  1. [Abstract and §4, Table 3] The abstract's claim that performance is 'not significantly or consistently affected by several aspects' is internally inconsistent with Table 3 and the surrounding text. Using the paper's own 2σ=0.03 threshold from §2.6, loss reweighting changes NLL by more than 0.03 on four of five corpora (acl 7.06→7.10, greek 6.70→6.88, latin 7.16→7.37, un 7.08→7.01), and the text explicitly states that 'Three out of the four corpora showing significant differences favor not reweighting the loss.' Reweighting is therefore significant in most corpora, even if the direction is not uniform. The abstract and §4 should be revised to name reweighting as an exception or to soften the significance language to 'small but not consistently directional effects.' As written, the central claim is not supported.
  2. [§2.6, Tables 2–7] The only variance estimate in the paper is based on 20 seeded runs on the ACL corpus, yielding 2σ=0.03, but every other corpus is reported as a single run per configuration. Applying this ACL-derived threshold to greek, latin, scifi, and un assumes that run-to-run variability is identical across corpora, which is not shown and is questionable given the large differences in corpus size and language. Consequently, statements such as 'None of the corpora show a significant difference' (Table 2) and the classification of differences as significant or not in Tables 3–7 are not supported for four of the five corpora. The authors should either provide per-corpus variance estimates (e.g., multiple seeds for at least the key comparisons) or explicitly present the 2σ value as an ACL-only heuristic and avoid strong significance claims elsewhere.
  3. [§3, Table 6] The claim that DETM is 'very stable when scaling vocabulary size' is also in tension with the reported numbers. On the UN corpus, NLL worsens from 7.10 at 20k to 7.17 at 80k, a change larger than the paper's own 2σ=0.03 threshold. While the other four corpora are indeed stable, the single-run nature of the UN measurement means this difference cannot simply be dismissed, and the blanket stability claim should be qualified with explicit attention to this outlier.
minor comments (4)
  1. [§2.6] The text says 'We format in bold the best performance in a table row unless it doesn't meet this threshold,' but none of the tables in the paper contain bold entries; either the formatting was omitted or the sentence should be removed.
  2. [§2.1] There is a typo: 'Leuven Databast of Ancient Books' should be 'Leuven Database of Ancient Books.'
  3. [§2.4] The phrase 'We have conveniently decided to focus on basic training and convergence' is informal and could be replaced with a neutral statement of what the paper does and does not claim about topic interpretability.
  4. [Tables and code availability] The references to ANONYMIZED dataset, code, and library URLs must be replaced with actual repository identifiers before publication, and Table 6's header 'V ocab Size' contains a stray space.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose test-set NLL and NPMI scores are computed from held-out data, and no central claim is derived from its own inputs.

full rationale

This paper is an empirical measurement study, not a derivation chain. The core quantities are per-word test-set NLL and NPMI coherence, both computed on held-out test documents (Sections 2.2 and 2.4), and the hyper-parameter sweeps vary one factor at a time against fixed defaults (Table 8). No parameter is fitted to the paper's own conclusions. The only estimated quantity, the 2-sigma = 0.03 significance heuristic, is obtained from 20 independently seeded ACL runs (Section 2.6) and then applied as a threshold to single-run comparisons on other corpora; this is a statistical robustness assumption, not a circular reduction. Even if the abstract's phrase 'not significantly or consistently affected' sits awkwardly with Table 3 under the paper's own threshold, that is an internal-consistency or correctness issue, not circularity. The self-citation to Sirin and Lippincott (2024) is used in Section 3 only as an illustrative formula for adverb topic-membership in scifi models and is not load-bearing for the central claims. There is no self-definitional step, no fitted input renamed as a prediction, no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result. The paper is self-contained against external benchmarks in the sense that its measurements are empirical and reproducible from the released data and code. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its conclusions rest on the empirical choices and assumptions listed above.

assumptions (4)
  • domain assumption NLL on held-out test subdocuments is a meaningful first-order measure of DETM training and convergence quality.
    Section 2.4 states NLL is used for basic training and convergence, while acknowledging it is not a proxy for human interpretability.
  • ad hoc to paper The empty-window smoothing policy (even mixture of nearest non-empty windows) does not distort the comparison across window counts.
    Section 2.5 introduces this interpolation to avoid division-by-zero; the paper argues it is sufficient, but this is not independently validated.
  • domain assumption The significance threshold estimated from 20 ACL runs transfers to the other corpora.
    Section 2.6 uses 2σ = 0.03 from ACL to decide which differences are significant elsewhere, despite single runs on other corpora.
  • domain assumption Word2Vec embeddings trained on train+development text provide appropriate input for DETM on all corpora.
    Section 2.2 describes embedding training; the paper does not evaluate sensitivity to embedding hyperparameters.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora." pith.science (2026). https://pith.science/paper/OGS6DQZ7

@misc{pith2026250419209,
  author       = {Pith},
  title        = {Pith review of: Dynamic Embedded Topic Models: properties and recommendations based on diverse corpora},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGS6DQZ7}},
  note         = {Machine review of arXiv:2504.19209}
}
read the original abstract

We measure the effects of several implementation choices for the Dynamic Embedded Topic Model, as applied to five distinct diachronic corpora, with the goal of isolating important decisions for its use and further development. We identify priorities that will maximize utility in applied scholarship, including the practical scalability of vocabulary size to best exploit the strengths of embedded representations, and more flexible modeling of intervals to accommodate the uneven temporal distributions of historical writing. Of similar importance, we find performance is not significantly or consistently affected by several aspects that otherwise limit the model's application or might consume the resources of a grid search.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Maria Antoniak. 2022. https://maria-antoniak.github.io/2022/07/27/topic-modeling-for-the-people.html Topic modeling for the people

    work page 2022

  4. [4]

    David Blei and John Lafferty. 2006 a . https://proceedings.neurips.cc/paper_files/paper/2005/file/9e82757e9a1c12cb710ad680db11f6f1-Paper.pdf Correlated topic models . Advances in neural information processing systems, 18:147

    work page 2006

  5. [5]

    David M Blei and John D Lafferty. 2006 b . https://dl.acm.org/doi/10.1145/1143844.1143859 Dynamic topic models . In Proceedings of the 23rd international conference on Machine learning, pages 113--120

    arXiv 2006

  6. [6]

    David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. https://dl.acm.org/doi/10.5555/944919.944937 Latent dirichlet allocation . Journal of machine Learning research, 3(Jan):993--1022

    arXiv 2003

  7. [7]

    Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, 30:31--40

    2009

  8. [8]

    Pierluigi Cassotti, Francesco Periti, Stefano De Pascale, Haim Dubossarsky, and Nina Tahmasebi. 2024. https://aclanthology.org/2024.eacl-tutorials.1/ Computational modeling of semantic change . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 1--8, St. Julian ' s, Mal...

    work page 2024

Show all 21 references
  1. [9]

    Chang, Jordan L

    Jonathan D. Chang, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. https://api.semanticscholar.org/CorpusID:215812433 Reading tea leaves: How humans interpret topic models . In Neural Information Processing Systems

    2009

  2. [10]

    Clarysse and M

    W. Clarysse and M. Depauw. 1996. https://www.trismegistos.org/ldab/ Leuven database of ancient books

    1996

  3. [11]

    Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2019. https://arxiv.org/abs/1907.05545 The dynamic embedded topic model . arXiv preprint arXiv:1907.05545

    2019 arXiv

  4. [12]

    Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020. https://arxiv.org/abs/1907.04907 Topic modeling in embedding spaces . Transactions of the Association for Computational Linguistics, 8:439--453

    2020 arXiv

  5. [13]

    first1kgreek. 2025. https://opengreekandlatin.github.io/First1KGreek/ First!kgreek project

    2025

  6. [14]

    Edouard Grave, Armand Joulin, Moustapha Ciss \' e , David Grangier, and Herv \' e J \' e gou. 2016. https://arxiv.org/abs/1609.04309 Efficient softmax approximation for gpus . CoRR, abs/1609.04309

    2016 arXiv

  7. [15]

    HathiTrust Foundation. 2023. https://www.hathitrust.org HathiTrust Digital Library

    2023

  8. [16]

    Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. 2021. https://arxiv.org/abs/2107.02173 Is automated topic model evaluation broken? the incoherence of coherence . Advances in neural information processing systems, 34:2018--2033

    2021 arXiv

  9. [17]

    Paul Pu Liang, Manzil Zaheer, Yuan Wang, and Amr Ahmed. 2020. https://arxiv.org/abs/2003.08197 Anchor & transform: Learning sparse representations of discrete objects . CoRR, abs/2003.08197

    2020 arXiv

  10. [18]

    Hongyuan Mei and Jason Eisner. 2017. https://arxiv.org/abs/1612.09328 The neural H awkes process: A neurally self-modulating multivariate point process . In Advances in Neural Information Processing Systems, Long Beach

    2017 arXiv

  11. [19]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. https://arxiv.org/abs/1301.3781 Efficient estimation of word representations in vector space . Preprint, arXiv:1301.3781

    2013 arXiv

  12. [20]

    Francisco J. R. Ruiz, Michalis K. Titsias, Adji B. Dieng, and David M. Blei. 2018. https://arxiv.org/abs/1802.04220 Augment and reduce: Stochastic inference for large categorical distributions . Preprint, arXiv:1802.04220

    2018 arXiv

  13. [21]

    Hale Sirin and Thomas Lippincott. 2024. https://aclanthology.org/2024.latechclfl-1.22/ Dynamic embedded topic models and change-point detection for exploring literary-historical hypotheses . In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultu...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.