The Hitchhiker's Guide to LDA

Chen Ma

REVIEW 3 major objections 5 minor 43 references

The Hitchhiker's Guide to LDA

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This Chinese book-length guide aims to hand-derive, step by step, every formula behind LDA's two inference methods — collapsed Gibbs sampling and variational EM — so that a beginner who has forgotten calculus can follow the entire…

desk verdict A competent Chinese-language re-derivation of standard LDA material; no new results, and the variational chapter contains a real conceptual error about mean-field factorization. read the letter →

arxiv 1908.03142 v2 pith:UDOCVOWM submitted 2019-08-07 cs.IR cs.CLcs.LG

Chen Ma This is my paper · ORCID

classification cs.IRcs.CLcs.LG MSC 62F1560J2268T50

keywords LatentDirichletAllocationtopicmodelscollapsedGibbssamplingvariationalinferenceEMDirichlet-multinomialconjugacymean-fieldapproximationparallel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This is a teaching book, not a research monograph: it sets out to make the mathematics of Latent Dirichlet Allocation fully accessible, promising in the preface and abstract that every formula is hand-derived step by step and that a reader who has forgotten calculus can still follow the whole chain. Its load-bearing claims are that the collapsed Gibbs sampling update (Eq. 3-20) and the variational EM updates (Eqs. 7-30 and 7-32) are derived completely and correctly from first principles, and that the reader can then implement both methods from the provided pseudocode and a line-by-line analysis of a reference C implementation. The book matters because LDA is one of the most widely used text models in industry and academia, and the gap between the standard research papers and a beginner's background is exactly what it tries to close. The book is written in Chinese and doubles as a practical handbook, with chapters on applications, two parallelization schemes, and a walkthrough of the reference code.

What carries the argument

For the Gibbs half, the carrying machinery is the Dirichlet–multinomial conjugacy pair: it makes the integrals over $\theta$ and $\varphi$ in Eqs. (3-16) and (3-18) collapse into ratios of products of gamma functions, and the factorial identity $\Gamma(x+1) = x\Gamma(x)$ then lets nearly every gamma cancel in the full conditional, leaving the two count ratios of Eq. (3-20). The book singles out this gamma cancellation as the central trick of the whole derivation. For the variational half, the carrying object is the mean-field factorization of Eq. (7-23), together with the Dirichlet expectation identity $E_q[\log \theta_i] = \Psi(\gamma_i) - \Psi(\sum_j \gamma_j)$; these convert the five-term lower-bound expansion of Eq. (7-27) into the two coupled update equations that are iterated in alternation until the bound converges.

What would settle it

Take two short documents, a vocabulary of a few words, and $K=2$ topics, enumerate the exact posterior $p(\mathbf{z}, \theta \mid \mathbf{w})$ by brute force over all $2^N$ topic assignments, and compare it with the collapsed-Gibbs sampler run from Eq. (3-20) and with the converged mean-field approximation from Eqs. (7-30) and (7-32); if the sampler's stationary distribution disagrees with the enumerated posterior beyond sampling noise, or if the variational updates oscillate instead of converging to a posterior close to the exact one, the respective derivation chain fails its promise.

Watch

Extended reading notes

Core claim

On the book's own terms, the central content is a complete, self-contained derivation chain for LDA's two inference methods. For the sampling route, the chain starts from the joint probability of the observed words and the latent topic assignments, integrates out the Dirichlet parameters $\varphi$ and $\theta$ — the move that gives 'collapsed' Gibbs sampling its name — and then uses the full-conditional distribution to arrive at Eq. (3-20), the sampling formula $$p(z_i = k \mid \mathbf{z}_{-i}, \mathbf{w}) \propto \frac{n_{k,w_i}^{(-i)} + \$\beta$}{$n_k^{{(-i)}}$ + V\$\beta$}\cdot\frac{n_{m,k}^{(-i)} + \$\alpha$}{$n_m^{{(-i)}}$ + K\$\alpha$},$$ where the gamma functions introduced in the derivation cancel out. For the optimization route, the chain builds the evidence lower bound, applies the mean-field factorization $q(\theta, \mathbf{z} \mid \gamma, \phi) = q(\theta \mid \gamma)q(\mathbf{z} \mid \phi)$ of Eq. (7-23), expands the bound into five terms, and differentiates to obtain the update equations $\phi_{n,i} \propto \beta_{i,w_n}\exp(\Psi(\gamma_i))$ and $\gamma_i = \alpha + \sum_n \phi_{n,i}$ (Eqs. 7-30, 7-32). The book's stated aim is that a reader who works through these chains ends with the standard published results, the ability to reproduce them unaided, and enough understanding to implement both algorithms.

Load-bearing premise

The variational route assumes that a document's topic assignments and its topic-proportion vector are independent under the approximating distribution; Section 7.4 concedes that they are not, so the expansions of $E[z_{n,i}\log\theta_i]$, $E[z_{n,i}\log\beta_{i,w_n}]$, and $E[z_{n,i}\log\phi_{n,i}]$ that produce updates (7-30) and (7-32) are approximations whose error is never quantified.

Editorial extensions

If this is right

A reader who follows Chapters 2 and 3 arrives at the collapsed Gibbs sampling formula (3-20) with the two algebraic moves — gamma cancellation and the cancellation of the product over topics and documents — made explicit, which most treatments omit.
The implementation chapter gives a working route from formula to code: the cumulative-method dice roll, the '-1, reassign topic, +1' update loop, and the memory estimate $O(KV)$ for the word-topic count array, so a beginner can implement LDA without assembling scattered sources.
The variational chapters reconstruct the original LDA paper's derivation with the compressed details filled in — the five-term expansion of the lower bound and the constrained optimizations for $\phi$ and $\gamma$ — making the classic variational EM method learnable.
The parallelization chapter explains why naive parallel Gibbs breaks the count statistics and where each fix (AD-LDA's global update, spark-LDA's row-and-column blocking) leaves residual error, equipping the reader to choose between the two schemes.
The applications chapter shows that the standard outputs of one LDA run (the $\theta$ and $\varphi$ matrices) can be repurposed, without model changes, for document similarity, tagging, recommendation features, topic ranking, and a quality-scoring heuristic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The book flags but never quantifies the mean-field approximation error; a natural extension is a tiny-corpus benchmark that measures the KL divergence between the converged variational posterior and a brute-force exact posterior, turning the caveat of Section 7.4 into a number.
The quality-scoring method of Section 4.2.6 — rating an article by its distance from the corpus-average vector — is illustrated with blog examples but no independent validation; a testable extension is to check it against a labeled spam-versus-curated dataset and to measure its correlation with human quality ratings.
The derivation chain the book lays out — write the joint, integrate out the conjugate priors, cancel gammas in the full conditional — is not specific to LDA, so the same pedagogical route could be reused to introduce other conjugate topic models such as HDP or supervised variants.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is not a research preprint; it is a book-length tutorial on LDA, originally published in Chinese, uploaded to arXiv. Second, the Gibbs-sampling half is mostly solid, but the variational half repeats a standard derivation and then explains the math incorrectly at one load-bearing point.

What it does well: the collapsed Gibbs derivation is careful and arrives at the standard update (3-20); the walkthrough of Blei's C code in Chapter 8 is genuinely useful; the book cites Blei et al., Griffiths and Steyvers, Heinrich, Newman et al., and Qiu, and its re-derivations benchmark against those sources rather than against itself. The author is also unusually open about limitations elsewhere, e.g., in the preface saying some proofs should be taken on trust.

Where it is soft: (1) There is nothing new. Every equation is a standard result. The application chapters (topic rank, word rank, quality score) are heuristics, with no code, data, or controlled evaluation; the informal accuracy and average-PV claims are unsupported. (2) The conceptual error: Section 7.4 tells the reader that the expansion of E_q[z log theta] into E_q[z] E_q[log theta] is approximate, because 'in reality z and theta are not independent.' That is wrong. Once you choose the factorized family q(theta,z)=q(theta)q(z) (Eq. 7-23), those expectations factor exactly. The approximation entering variational inference is the replacement of p by q; the algebra under q is exact. A beginner reading Section 7.4 will be told a mathematically correct step is approximate, undercutting the book's stated promise of rigorous step-by-step derivation. (3) The arXiv PDF has badly OCR-corrupted equations, which made fine-grained checking harder; I would not judge individual formulas from the scan alone.

For a reader: a motivated Chinese-speaking beginner wanting a hand-held walkthrough of LDA could get value from the Gibbs chapters and the C-code chapter. It is not a research contribution and should not be peer-reviewed as one. If it were to be revised as a book, the Section 7.4 explanation should be corrected.

Referee Report

3 major / 5 minor

Summary. This is a Chinese-language, book-length tutorial manuscript (eight chapters plus front matter) on Latent Dirichlet Allocation. Its stated purpose is pedagogical: the preface promises that every formula is hand-derived step by step so that a beginner, even one who has forgotten calculus, can follow, and the abstract promises a rigorous mathematical derivation of the two standard inference approaches. Chapter 2 develops prerequisite probability material (gamma, Beta, Dirichlet, conjugacy); Chapter 3 derives collapsed Gibbs sampling and arrives at the standard update (Eq. 3-20) together with the posterior estimates (3-21)-(3-22); Chapter 4 presents six applications, including topic/word ranking and an article-quality scoring method; Chapter 5 explains AD-LDA and a block-structured Spark implementation; Chapters 6-8 give background on variational calculus and a full derivation of variational EM for LDA, ending with the updates (7-30) and (7-32), the M-step estimators (7-36) and (7-38), and a detailed walkthrough of Blei's lda-c code. The book claims no new theorem, estimator, or dataset result; its contribution is expository.

Significance. As an exposition, the book is largely successful where it reproduces standard results: the collapsed Gibbs derivation in Sec. 3.4 is correct in substance, and the explanation of the Gamma-function cancellations and product cancellations is genuinely instructive; the variational derivation reaches the correct ELBO (7-22) and the standard coordinate-ascent and M-step formulas; and Chapter 8's line-by-line reading of lda-c, including the log-sum trick and the incremental gamma update (8-3), is a real strength that will help practitioners map mathematics onto code. The parallelization chapter gives a fair conceptual account of AD-LDA and of the Spark method described in Sec. 5.2. If the exposition were fully corrected, the book would be a useful bridge for Chinese-speaking beginners. It is not a research contribution, however, so its significance must be judged on accuracy and clarity rather than novelty; in that judgment, the conceptual explanation in the variational half and the empirical claims in the application chapters currently fall short of the book's own stated standard.

major comments (3)

[Sec. 7.4, Eqs. (7-22)-(7-23), (7-30), (7-32)] Section 7.4's 'error discussion' mislocates the approximation in the variational derivation. The three expansions that Section 7.4 labels as approximate - E_q[z_{n,i} log theta_i] = E_q[z_{n,i}] E_q[log theta_i], E_q[z_{n,i} log beta_{i,w_n}] = phi_{n,i} log beta_{i,w_n}, and E_q[z_{n,i} log phi_{n,i}] = phi_{n,i} log phi_{n,i}, used in the five-term expansion (7-27) - are exact under the variational family q(theta,z|gamma,phi) = q(theta|gamma) prod_n q(z_n|phi_n) introduced in Eq. (7-23), because the factorized q makes z_{n,i} depend only on phi_n and log theta_i only on gamma. The approximation in variational inference consists entirely in replacing the true posterior p(theta,z|w,alpha,beta) with q; that is what the KL term in (7-20)-(7-21) measures, and no further approximation is introduced when the ELBO is expanded under q. By telling the reader that the expansions 'contain errors' because 'in reality z and theta are not independent,' Section 7.4 conflates the true posterior with the variational distribution and teaches a beginner that an exact algebraic step is approximate. Because the manuscript's central claim is rigorous step-by-step derivation, this passage should be rewritten so that the mean-field factorization (7-23) is identified as the single approximation and the expansions are presented as exact consequences of it.
[Secs. 4.2.5-4.2.6, Table 4-5; Sec. 5.2.4] Several quantitative claims in the application and parallelization chapters are unsupported, and one is contradicted by the book's own table. (a) Section 4.2.5 asserts that word rank '平均可以达到80%的准确度' (can reach 80% accuracy on average) with no definition of accuracy, no test corpus, and no evaluation protocol. (b) Section 4.2.6's article-quality algorithm rests on the assumption that distance from the corpus-average 'god vector' (Eqs. 4-4 and 4-5) measures quality; the assumption is never validated against human ratings or any independent quality signal, and the demonstration consists of the author's inspection of twenty blog posts. (c) The text claims, on the basis of Table 4-5 and Fig. 4-14, that higher quality scores are associated with higher average page views, but the table is non-monotonic at the top end: the 0.9-1.0 bin has mean PV 449.8, below the 0.8-0.9 bin's 588.3, and the 0.5-0.6 bin's 167.4 exceeds the 0.4-0.5 bin's 149.3; no confidence intervals or bin-size corrections are given, and the top bin contains only 390 articles versus 2975 in the second bin. (d) Section 5.2.4 asserts that experiments show the distributed method's perplexity matches single-machine Gibbs Sampling, but no experiment is reported in the chapter. These claims should be either properly supported or explicitly presented as anecdotal illustrations from the author's practice.
[Sec. 2.5, Eq. (2-11); Preface] Section 2.5 leaves the proof of the Dirichlet normalization identity (2-11) 'to the reader' ('这个证明留给读者来完成'), which contradicts the preface's explicit promises that Chapter 2 '力求做到关键证明不遗漏' (strives to omit no key proof) and that every formula is hand-derived ('每一条公式都由作者手把手为您推理'). This is not a cosmetic gap: Eq. (2-11) is invoked at load-bearing steps of the Gibbs derivation, Eqs. (3-16), (3-18), and (3-19), where integrals over phi and theta are collapsed into Delta functions. Since the analogous Beta-function identity (2-6) is proved twice in Section 2.3, supplying the Dirichlet version by the same substitution argument would be straightforward; alternatively, the preface's promise should be reworded.

minor comments (5)

[Eqs. (3-20), (4-5), (7-27), (7-38)] Throughout the manuscript, displayed equations contain stray symbols and interleaved annotations (for example, the in-line commentary inside the derivation of Eq. (3-20) and the corrupted characters in Eqs. (4-5), (7-27), and (7-38)). For the target beginner readership, a clean typeset pass with numbered, self-contained displays is needed.
[Sec. 4.2.6, Fig. 4-14] Figure 4-14 plots bin-averaged PV against score, but the bin sizes differ by an order of magnitude (390 versus 2975 articles in the top two bins); the apparent trend should be shown with bin counts and dispersion, and the axes and units should be labeled.
[Preface] The preface's claim that the book '首次将 LDA 引入大数据时代' (first brings LDA into the big-data era) is not supported by Chapter 5, which surveys existing methods such as AD-LDA and the Spark method it describes; this claim should be removed or substantially softened.
[Sec. 3.2 vs Sec. 7.1, Table 7-1] The two halves of the book use different notation for the topic-word distribution (a Dirichlet-sampled phi_k in Chapter 3 versus a fixed matrix beta in Chapter 7). Table 7-1 documents the change, but a forward reference at Section 3.2 would save beginners real confusion.
[References] The reference list mixes standard sources (Blei et al. 2003; Griffiths and Steyvers; Heinrich) with informal web pages and blog posts, and the 'further reading' list at the end repeats earlier chapter lists; a single consolidated bibliography with formal citations would serve the book's purpose better.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LDA derivations are self-contained re-derivations benchmarked against external sources, not fitted inputs renamed as predictions.

full rationale

The book's central claims are pedagogical re-derivations of standard LDA results. The collapsed Gibbs sampling formula (Eq. 3-20) is obtained by writing the joint distribution (Eqs. 3-13, 3-19), integrating out the Dirichlet parameters via the Dirichlet-multinomial conjugacy relation (Eq. 2-11), and simplifying the full conditional; the result matches the external derivations of Heinrich and of Griffiths and Steyvers rather than any result of the author's own prior work. The variational derivation in Section 7.3 likewise starts from an explicit mean-field factorization (Eq. 7-23), expands the five terms of the ELBO (Eq. 7-27), and optimizes the lower bound to obtain the standard updates (Eqs. 7-30, 7-32, 7-36, 7-38); these are derived from the objective rather than assumed. No fitted parameter is later renamed as a prediction, and the Chapter 4 applications are heuristic demonstrations on examples, not fitted-then-predicted quantities. The book contains no load-bearing self-citations and invokes no uniqueness theorem from the author's own prior work. The only notable flaw is conceptual rather than circular: Section 7.4 describes the expectations under the factorized q as containing additional approximation errors because 'in reality z and theta are not independent,' but under the book's own mean-field assumption (Eq. 7-23) the factorizations E_q[z_{n,i} log theta_i] = E_q[z_{n,i}] E_q[log theta_i] are exact; the approximation enters only when p is replaced by q. This is a correctness or exposition issue about where the variational approximation occurs, not a circularity that reduces the derivation to its inputs. Accordingly, no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The book's derivations rest on standard mathematics (conjugacy, MCMC theory, Jensen's inequality) that it either proves or cites. The one genuinely ad hoc postulate is the quality-scoring premise in Section 4.2.6, and the variational derivation rests on an approximation the book itself admits in Section 7.4. No free parameters are fitted to data by the book; the only hand-chosen numbers are the unreported weights in the topic-rank demonstration.

free parameters (1)

topic rank weighting coefficients a and b = not reported
Section 4.2.4 defines final_score = a * score_topic->doc + b * score_topic->word but never reports the values used to produce the ranking screenshots, so the demonstrated ranking is not reproducible.

assumptions (4)

standard math Dirichlet is the conjugate prior of the multinomial (Dirichlet-multinomial conjugacy).
Proved in Section 2.6 and used throughout Section 3.4 (Eqs. 3-16, 3-18) and Section 7.3 to integrate out theta and phi.
domain assumption The mean-field factorization q(theta, z | gamma, phi) = q(theta | gamma) q(z | phi) is a benign approximation.
Introduced at Eq. (7-23) in Section 7.3.3. Section 7.4 explicitly concedes that z and theta are not independent, making this an acknowledged approximation on which the Section 7 derivation rests.
standard math Markov chains converge to a unique stationary distribution under detailed balance and ergodicity.
Sections 3.3.1 and 3.3.2 rely on this to justify Gibbs sampling; stated as standard MCMC theory with cited references.
ad hoc to paper Proximity of a document's probability vector to the corpus-average 'god vector' measures article quality.
Section 4.2.6, Eq. (4-4) defines the reference vector and asserts that closeness to it indicates quality; no independent validation is provided beyond the author's own examples.

invented entities (1)

Mass article probability vector (众文档平均概率向量, also called the god vector)
purpose: Reference vector in Section 4.2.6 whose distance from a document's topic-word probability vector defines the article quality score.
Defined by Eq. (4-4) from corpus statistics; the claim that closeness to it indicates writing quality has no falsifiable handle outside the book's own examples and the informal PV table.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Hitchhiker's Guide to LDA." pith.science (2026). https://pith.science/paper/UDOCVOWM

@misc{pith2026190803142,
  author       = {Pith},
  title        = {Pith review of: The Hitchhiker's Guide to LDA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UDOCVOWM}},
  note         = {Machine review of arXiv:1908.03142}
}

read the original abstract

Latent Dirichlet Allocation (LDA) model is a famous model in the topic model field, it has been studied for years due to its extensive application value in industry and academia. However, the mathematical derivation of LDA model is challenging and difficult, which makes it difficult for the beginners to learn. To help the beginners in learning LDA, this book analyzes the mathematical derivation of LDA in detail, and it also introduces all the knowledge background to make it easy for beginners to understand. Thus, this book contains the author's unique insights. It should be noted that this book is written in Chinese.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages

[2]

大力神海格力斯式的笨重劳动

Zhuolin Qiu. Gibbs Collapsed Sampling for Latent Dirichlet Allocation on Spark. JMLR W&CP 36 : 17–28, 2014 68 第6章变分贝叶斯的启蒙在我们的故事叙述到这里的时候，读者已经发现，Gibbs Sampling 技术虽然实现简单且输出效果不错，但是其采样的效率犹如“大力神海格力斯式的笨重劳动” （欧拉语）。因此有必要看看从另一种思路出发而产生的技术，变分贝叶斯法（Variational Bayes）推导是 Blei 的 LDA 论文原作中的经典方法，如果说学习 LDA 模型，但没有学过原作中的 VB 法推导，仍然称不上学会了 LDA。这种方法运行速度比 Gibbs Sampling 方法...

work page 2014
[3]

先来观察 Y=g(x)，由于 g(x)属于严格上升函数，因此分布函数换成反函数 X=h(y)表达： 换成表达        X ( ) ( ) ( ( ) ) ( h( )) ( ) h y P Y y P g X y P X y f t dt P 关于 y 求导数，得到 y 的密度函数( ) ( ( )) ( )l y f h y h y

work page
[4]

充分统计量 u(x)关于极大似然估计值 MLE的期望

又因为 h 是 g 的反函数，则 h(g(x))=x(此式左右两边同时对 x 求导)，推理出=>    ( ( )) ( ) 1h g x g x 即    1( ( )) ( )h g x g x 因此         ( ( )) ( ) ( ( )) ( )E g X y l y dy y f h y h y dy ，把 y 换成 g(x),则      1( ( ( ))) ( ); ( ( )) ; ( )( )f h g x f x h g x dy g x dx g x 最后           ( )( ( )) ( ) ( ) ( ) ( )( )f f xE g X g x g x dx g x f x...

work page
[5]

基于变分贝叶斯方法的医学图像分割

华俊豪. 基于变分贝叶斯方法的医学图像分割. 浙江工业大学本科毕业设计 2013

work page 2013
[6]

7.2 Calculus of Variations

Gilbert Strang. 7.2 Calculus of Variations. 2006

work page 2006
[7]

Aniket C. Aranake. Notes on the Calculus of Variations and Lagrange Multipliers. 2011.02.13

work page 2011
[8]

Wainwright, Michael I

Martin J. Wainwright, Michael I. Jordan. Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends in Machine Learning. 2008:1- 305

work page 2008
[9]

Variational Inference(讲义)

David M.Blei. Variational Inference(讲义)

work page

Show all 43 references

[10]

Life After the EM Algorithm: The Variational Approximation for Bayesian Inference

Dimitris Tzikas, Aristidis Likas, Nikolaos Galatsanos. Life After the EM Algorithm: The Variational Approximation for Bayesian Inference
[11]

Pattern Recognition and Machine Learning

Christopher M.Bishop. Pattern Recognition and Machine Learning. Springer. 2006:Chapter 10(Approximate Inference)& p.490

2006
[12]

机械工业出版社2009.03:p.673 – 678

Joel Hass, Maurice D.Weir, George B.Thomas, Jr.李伯民译托马斯大学微积分. 机械工业出版社2009.03:p.673 – 678
[13]

A wise man never knows all, only fools know everything

陈希孺. 概率论与数理统计. 中国科学技术大学出版社. 2009.2.1. p.103,p.110 94 第7章 LDA 的变分贝叶斯法如果说 Collapsed Gibbs Sampling 版本的 LDA 犹如《新约圣经》，那么 Variational Bayes 法推导的 LDA 就像《旧约圣经》一样，如果说一个人学习 LDA 却不知道 LDA 的变分推断方法，就像一个基督徒不读《圣经》一样，这仍然称不上了解 LDA，溯本求源是我们应有的态度。LDA 的变分贝叶斯法既是 LDA 模型的起点，也是一个启发新技术的契机。 7.1 Late...

2009
[14]

选取文档长度（文档中的单词个数）~ ( )N possion 
[15]

选取文档中的主题分布 ~ ( )Dir  33 主要推导部分在第5节和附录(Appendix)部分 95 如前面章节所述，这里的是 Dirichlet 分布的超参数，一共维（而 是设置的主题总数），而  0 1 1( , ,..., ) TK    是产生该文章主题的多项分布的概率参数，其中i是第i个主题被选择的概率，从 Dirichlet 产生参数后，再用i去产生具体主题z。 3．上面两步完成后，进而产生一篇文章 N 个词里的每个词： For each word nw in 1 ~N words: (...
[16]

选取文档的主题分布 ~ ( )Dir 
[17]

接近”这个概念如何在具体计算中体现，也就是说如何衡量两个概率分布的相似度，幸运的是，这已经有了现成的工具：通常使用KL距离。稍做一点简单的推导，就会发现“优化变分下界

For each word nw of N words: a) 从概率  ( | , )p w   中选取一个单词nw 这个过程也就将文档边缘分布以连续混合分布（continuous mixture distribution）定义出来36：                 1 ( | , ) ( | ) ( | , ) N n n p w p p w d       (7-10) 可以发现公式(7-8)实际就是公式(7-9)代入公式(7-10)之后的结果。公式 (7-10)中...

1983
[18]

maximize wrt. ,n i 先来研究,n i ，也就是第 n 个单词是被第 i 个潜在 topic 产生的概率。应该特别注意到一个词在不同主题的概率之和等于 1，即限制条件：   , 1 1 K n i i  (7-28) 注意该限制条件与 Gibbs 版本的归一化，即第 3 章的公式(3-22)不同。将变分下界公式(7-27)只留下有关,n i 的项，令,i v 为令  ( 1 | 1)v inp w z 的合适的那个单词 v，也就是令1vnw 的 v。由于存在限制条件(7-28)，利用拉格朗日乘数法求解...
[19]

maximize wrt. i：现在轮到i参数，其中{1,..., }i K ，下界函数只留下i有关的项：                                            [ ] , 1 1 1 1 1 1 1 1 1 ( 1)( ( ) ( ))+ ( ) ( ) log ( ) log ( ) ( 1) ( ( ) ( )) i K K N K K i i j n i i j i j n i j K K K K i...
[20]

Latent Dirichlet Allocation

David M.Blei & Andrew Y .Ng. Latent Dirichlet Allocation
[21]

Spring 2013

Wayne Xin Zhao(batmanfly@gmail.com) Note1: Variational Methods for Latent Dirichlet Allocation. Spring 2013

2013
[22]

Variational Inference for LDA

Zhao Zhou. Variational Inference for LDA. The Hong Kong University of Science and Technology
[23]

Latent Dirichlet Allocation: Towards a Deeper Understanding

Colorado Reed. Latent Dirichlet Allocation: Towards a Deeper Understanding
[24]

Exchangeable Sequences, Polya’s Urn and De-Finetti’s Theorem

Mohammad Emtiyaz Khan. Exchangeable Sequences, Polya’s Urn and De-Finetti’s Theorem. CS,UBC. October 6, 2007

2007
[25]

BERNARDO

JOS´E M. BERNARDO. The Concept of Exchangeability and its Applications. Bhattacharya Memorial Volume. December 9, 1996

1996
[26]

hello world

David M.Blei. Variational Inference(讲义) 参考文献 113 第8章 LDA 变分 EM 实现对于实干家来说，来到了令人兴奋的一章，这一章将系统性地介绍 Blei 的 LDA 的 C 语言源代码的实现。回忆起第一次写程序的情形，运行一段代码，兴奋地看到“hello world”出现在闪闪发光的绿色屏幕，本章也将令人兴奋的 LDA 变分 EM 搬上电脑屏幕，这也算是对前文漫长的推导的一个补偿。如果初看 Blei 版本的源代码，初学者往往会迷惑于代码中一些精妙的地方。所以这一章的安排是这样的：首先根据前文推导...
[27]

下界” 。 [em max iter] 变分 EM 的是最大的迭代次数。 [em convergence] 变分 EM 的收敛条件。(score_old - score) / abs(score_old)如果小于这个数值，则停止迭代（在迭代超过最大迭代次数之后）。请注意这个参数是整个语料集的似然函数的“下界

<iteration>.beta 包含主题分布的 log 对数，每一行代表一个主题；在第 k 行，每一个元素是 log(p(w|z=k)) 另外有两个文件将会被创建：[name].gamma 是每篇文章的变分狄利克雷参数（variational Dirichlet parameters），也即 doctopic 分布的矩阵文件，每一行是一个 doc，而每一列是一个 topic 的概率；[name].likelihood 则是每篇文章的似然函数的下界。 3.配置参数配置参数配置放置在 settings.txt 文件中，共有以下设置项： ...
[28]

Gibbs Sampling for the Uninitiated

Philip Resnik and Eric Hardisty. Gibbs Sampling for the Uninitiated. CS-TR-4956 April 2010

2010
[29]

Parameter estimation for text analysis

Gregor Heinrich. Parameter estimation for text analysis
[30]

LDA数学八卦 2013.2.8

靳志辉(Rickjin). LDA数学八卦 2013.2.8

2013
[31]

概率论与数理统计

陈希孺. 概率论与数理统计. 中国科学技术大学出版社. 2009.2.1

2009
[32]

Latent Dirichlet Allocation

David M.Blei & Andrew Y.Ng. Latent Dirichlet Allocation
[33]

Reading Note : Parameter estimation for text analysis 暨LDA学习小结

持之以恒. Reading Note : Parameter estimation for text analysis 暨LDA学习小结. http://www.xperseverance.net/blogs/tag/lda/ . 2013.3.5

2013
[34]

Christopher M. Bishop. Pattern Recognition And Machine Learning. Springer. 2007.10.1. chapter 2 & chapter 11

2007
[35]

Probabilistic Topic Models

Mark Steyvers. Probabilistic Topic Models
[36]

Lafferty

David M.Blei & John D. Lafferty. Topic Models
[37]

Topic Signiﬁcance Ranking of LDA Generative Models

Loulwah AlSumait and Daniel Barbara,James Gentle,Carlotta Domeniconi. Topic Signiﬁcance Ranking of LDA Generative Models
[38]

Gibbs Collapsed Sampling for Latent Dirichlet Allocation on Spark

Zhuolin Qiu. Gibbs Collapsed Sampling for Latent Dirichlet Allocation on Spark. JMLR W&CP 36 : 17–28, 2014

2014
[39]

Distributed Inference for Latent Dirichlet Allocation

David Newman, Arthur Asuncion, Padhraic Smyth, Max Welling. Distributed Inference for Latent Dirichlet Allocation. {newman,asuncion,smyth,welling}@ics.uci.edu
[40]

最大似然估计 (Maximum likelihood estimation) http://www.cnblogs.com/liliu/archive/2010/11/22/1883702.html

2010
[41]

GibbsLDA++ Reference

Xuan-Hieu Phan and Cam-Tu Nguyen. GibbsLDA++ Reference. http://gibbslda.sourceforge.net/
[42]

An Introduction to MCMC for Machine Learning

Christophe Andrieu, Nando de Freitas, Arnaud Doucet, Michael I.Jordan. An Introduction to MCMC for Machine Learning. Kluwer Academic Publishers. 2001.9.10

2001
[43]

Visualizing Topic Models

Allison J.B.Chaney and David M.Blei. Visualizing Topic Models. Association for the Advancement of Artiﬁcial Intelligence (www.aaai.org). 2012

2012
[44]

Peacock：大规模主题模型及其在腾讯业务中的应用

赵学敏王莉峰王流斌孙振龙严浩靳志辉王益. Peacock：大规模主题模型及其在腾讯业务中的应用

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [2]

大力神海格力斯式的笨重劳动

Zhuolin Qiu. Gibbs Collapsed Sampling for Latent Dirichlet Allocation on Spark. JMLR W&CP 36 : 17–28, 2014 68 第6章变分贝叶斯的启蒙在我们的故事叙述到这里的时候，读者已经发现，Gibbs Sampling 技术虽然实现简单且输出效果不错，但是其采样的效率犹如“大力神海格力斯式的笨重劳动” （欧拉语）。因此有必要看看从另一种思路出发而产生的技术，变分贝叶斯法（Variational Bayes）推导是 Blei 的 LDA 论文原作中的经典方法，如果说学习 LDA 模型，但没有学过原作中的 VB 法推导，仍然称不上学会了 LDA。这种方法运行速度比 Gibbs Sampling 方法...

work page 2014

[2] [3]

先来观察 Y=g(x)，由于 g(x)属于严格上升函数，因此分布函数换成反函数 X=h(y)表达： 换成表达        X ( ) ( ) ( ( ) ) ( h( )) ( ) h y P Y y P g X y P X y f t dt P 关于 y 求导数，得到 y 的密度函数( ) ( ( )) ( )l y f h y h y

work page

[3] [4]

充分统计量 u(x)关于极大似然估计值 MLE的期望

又因为 h 是 g 的反函数，则 h(g(x))=x(此式左右两边同时对 x 求导)，推理出=>    ( ( )) ( ) 1h g x g x 即    1( ( )) ( )h g x g x 因此         ( ( )) ( ) ( ( )) ( )E g X y l y dy y f h y h y dy ，把 y 换成 g(x),则      1( ( ( ))) ( ); ( ( )) ; ( )( )f h g x f x h g x dy g x dx g x 最后           ( )( ( )) ( ) ( ) ( ) ( )( )f f xE g X g x g x dx g x f x...

work page

[4] [5]

基于变分贝叶斯方法的医学图像分割

华俊豪. 基于变分贝叶斯方法的医学图像分割. 浙江工业大学本科毕业设计 2013

work page 2013

[5] [6]

7.2 Calculus of Variations

Gilbert Strang. 7.2 Calculus of Variations. 2006

work page 2006

[6] [7]

Aniket C. Aranake. Notes on the Calculus of Variations and Lagrange Multipliers. 2011.02.13

work page 2011

[7] [8]

Wainwright, Michael I

Martin J. Wainwright, Michael I. Jordan. Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends in Machine Learning. 2008:1- 305

work page 2008

[8] [9]

Variational Inference(讲义)

David M.Blei. Variational Inference(讲义)

work page