REVIEW 4 major objections 4 minor 17 references
Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Running LDA on language models' abstractive retellings of literary passages yields more precise and informative topics than running LDA on the original text or asking the models to list topics directly.
desk verdict Useful, honest method paper on LM retellings for literary topic modeling, but the study-guide contamination worry is real and should be addressed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is abstractive retelling: a single prompt tells a language model to 'describe', 'summarize', or 'paraphrase' a 250-token book passage 'for a literary scholar analyzing narrative content,' and LDA is fit to the resulting one-paragraph retellings after light preprocessing (lowercasing, removing character names, filtering frequent and rare words). The prompt turns 'showing' into 'telling'—low-level sensory detail into high-level conceptual exposition—which is exactly the representation LDA's bag-of-words assumptions need. The verb choice is the control: describe and summarize encourage abstraction and outperform paraphrase, which stays closer to the surface form.
What would settle it
Build a labeled dataset in which each passage's surrounding paragraphs are manually checked to share the theme of the quote's tag, then rerun the comparison of Retell, default LDA, and TopicGPT-lite; if Retell's advantage shrinks or disappears on this clean set, the reported gains are an artifact of label-context alignment.
Extended reading notes
Core claim
The paper's central claim is that the bottleneck in topic modeling literature is not the topic model but the representation it sees: narrative prose encodes themes implicitly, and LDA's bag-of-words assumptions fail on that surface form. Retell removes the bottleneck by having an instruction-tuned language model 'tell' what a passage 'shows'—with describe, summarize, or paraphrase prompts—and then fitting LDA to those retellings. Across four small LMs, Retell's most prominent topics are judged by crowdworkers to be 'very related' to gold themes far more often than default LDA's topics are (averaging 0.59–0.60 across models versus much lower baselines), and more informative than TopicGPT-lite's often over-broad labels such as 'human nature' or 'life'. In a case study on racial/cultural identity in U.S. high-school English books, the probability of Retell's race-related topics rises for passages that human annotators coded as mentioning or discussing race, and combining two relevant topics recovers more such passages than default LDA's single topic without losing precision.
Load-bearing premise
The evaluation assumes that the paragraph context preceding a reader-tagged or study-guide-tagged quote is about the same theme as the tag attached to the quote; if the surrounding passage is about something else, the gold labels used to score every method are unreliable and all comparisons inherit that noise.
Editorial extensions
If this is right
- Scholars working with literary corpora can surface cross-cutting themes with a single LM prompt followed by LDA, without large API budgets.
- Retell topics are more interpretable because they use abstractive terms (e.g., 'financial', 'racial') rather than surface words (e.g., 'money', 'black'), making them easier to relate to human notions of theme.
- The method can screen collections for passages of interest: in the race case study, combining two relevant topics recalled more expert-annotated passages than default LDA without sacrificing precision.
- Because topic granularity k is set after retellings are produced, researchers can adjust k without rerunning the LM.
- Direct LM topic labeling with small models tends to collapse into a few overly broad labels; Retell avoids this by keeping LDA's distributional structure.
Reading between the lines
- If the gain comes from abstraction, Retell could be combined with other lexical or embedding methods (e.g., clustering or keyword extraction) beyond LDA, though the paper only tests LDA.
- The error analysis hints that retellings can leak book-level knowledge from pretraining, so Retell may work better on canonical or widely discussed books than on obscure ones; a comparison across familiarity would test this.
- The method's premise suggests it will help most in genres dense with sensory detail (literary fiction) and less in expository prose, where paraphrase adds little abstraction; a genre-by-genre evaluation would quantify this.
- For cultural analytics, Retell's topic probabilities could serve as a preliminary screening filter to direct close reading toward candidate passages, reducing the 'needle-in-haystack' annotation burden the authors describe.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Retell, a topic modeling pipeline that applies LDA to large language models' abstractive retellings of literary passages rather than to the original text. It compares Retell across four LMs and three retelling verbs against two baselines: LDA on the original passages and TopicGPT-lite, a simplified version of TopicGPT that directly elicits topic labels. Evaluation proceeds through crowdworker ratings of whether a method's most prominent topic is related to scraped literary-theme labels, a passage-level topic relevance/intrusion study, and a case study on racial/cultural identity passages in books taught in U.S. high schools. The paper reports that Retell outperforms both baselines on theme relatedness and identifies more race-relevant passages in the case study, and it releases code.
Significance. If the result holds, Retell offers a low-cost, easy-to-use method for humanities researchers who want to surface cross-cutting themes from literary text with smaller LMs. The paper is strong on transparency: it releases code, tests multiple models and retelling verbs, includes human annotation in two evaluation modes, and accompanies the case study with a detailed codebook and explicit limitations. Its main weakness is that the empirical evidence for the stated mechanism is undercut by the lack of any control for pretraining contamination from the very study-guide sources used as gold labels, and by the absence of uncertainty measures in the headline evaluation. The method is promising but the central claim, as currently worded, is broader than the evidence supports.
major comments (4)
- [§4.1, §6.2.2] The evaluation's gold labels are scraped from Goodreads, SparkNotes, and LitCharts (§4.1), all of which are plausible components of the pretraining data for the tested LMs. The paper provides no control for this overlap, so the observed advantage of Retell over default LDA and TopicGPT-lite could arise from the LM reproducing memorized study-guide themes rather than from abstractive retelling of the passage. The paper's own error analysis in §6.2.2 documents exactly this mechanism: a retelling includes a character's Iranian background that is absent from the passage, which the authors attribute to the LM's familiarity with the book. This is load-bearing because the central claim is about the method's ability to translate surface form into higher-level concepts; without a control using post-cutoff or obscure texts, or a test that removes identifying book-level information, the mechanism and the method's generality to unpublished or niche works remain unverified.
- [Table 2, Table 8] The headline result in Table 2 (and the extended version in Table 8) is reported as rating proportions without any uncertainty quantification. No confidence intervals, standard errors, or significance tests are given for the comparisons across methods or across values of k. This makes the central claim that Retell consistently outperforms baselines difficult to evaluate, especially since some differences (e.g., certain Retell variants versus TopicGPT-lite) are numerically small. The authors should report bootstrap confidence intervals over the 60 passage sets and appropriate paired tests, and apply the same rigor to the precision/recall comparisons in Appendix C.1.
- [Abstract, §5, Appendix C.4] The abstract and Section 5 state the central claim without the resource-efficiency qualifier that the evidence actually supports. Appendix C.4 shows that with GPT-4o, TopicGPT-lite achieves similar crowd ratings to Retell-describe and higher precision and recall (Table 10). The conclusion's phrasing 'especially for resource-efficient, small LMs' is accurate, but the abstract and Section 5 should be revised so that the general claim is scoped to smaller LMs, or additional evidence should be provided for larger LMs.
- [§4.1] The evaluation assumes that the paragraph context preceding a Goodreads, SparkNotes, or LitCharts quote does not deviate far from the tag or theme attached to the quote. This assumption is stated but not validated, despite a quote-matching rate of only 68.0 and the inclusion of up to 250 tokens of preceding context. If the surrounding passage differs thematically from the quote, the gold labels are noisy for every method, and all comparisons in Table 2 inherit that noise. The authors should either validate the assumption on a sample (e.g., human ratings of whether the full passage matches the quote's theme) or restrict evaluation to passages where the quote is representative of the whole window. The manual recoding of scraped tags/themes into 27 topics also lacks inter-annotator reliability, which is important because this recoding defines the gold set.
minor comments (4)
- [Table 2] The column structure of Table 2 is difficult to parse; the 'Rating /k' row lists many numbers without visually separating model/k combinations, and the repeated '✓', '?', and '✗' rows make it hard to map ratings to methods. A grouped layout or a separate table per model would improve readability.
- [Table 3] The note that ratings 'differ significantly for each method' is ambiguous; please specify which pairwise comparisons are significant (e.g., Topic 1 versus Intruder) and whether any multiple-comparison correction was applied.
- [§5] The phrase 'different instructive verbs' appears to be a slip for 'retelling verbs' or 'instruction verbs'; the current wording is imprecise.
- [Appendix C.1, Table 9] The absolute precision values in Table 9 are low (e.g., 0.128 for Retell-summarize at k=50); the authors should report the base rate of positive passage pairs so that readers can judge the effect size, and should comment on whether precision at these levels is practically useful for exploratory literary analysis.
Circularity Check
No significant circularity: Retell is an unsupervised pipeline evaluated against external gold labels; the minor self-citations provide data or error-analysis context but do not carry the central claim.
full rationale
The paper's central claim is an empirical comparison: LDA is fit to language-model retellings of passages using default decoding parameters, with no parameter tuned to the gold labels or human ratings (§3.1, §5). The pipeline is therefore not self-definitional or a fitted-input-called-prediction case: the retellings are generated from the passage alone, and the evaluation labels come from external sources (§4.1). The few self-citations (Lucy et al. 2025 for the ELA booklist, Levine et al. 2021 for teacher-listed books, and Chang et al. 2023 in the error analysis) supply data or an explanation for memorized book context, but the main topic-modeling comparison does not reduce to those citations. The Limitations section explicitly acknowledges that LMs can reproduce book-level knowledge from pretraining and that online gold labels carry content-production bias; these are real validity threats, but they are not circular reductions, because the method's output is not defined in terms of the gold labels and no fitted parameter is renamed as a prediction. No circular step is exhibited.
Assumptions & free parameters
free parameters (3)
- LDA topic count k =
50 (also 100, 200, or TopicGPT-determined values 14, 89, 157, 206, 538)
- Passage context window =
250 whitespace tokens
- Case-study keyword sampling threshold =
PMI >= 0.15
assumptions (4)
- domain assumption The paragraph preceding a quoted theme label has the same theme as the label itself.
- domain assumption LM retellings are faithful enough that LDA topics over them preserve, rather than distort, the thematic content of passages.
- domain assumption Crowdworker ratings of semantic relatedness between a predicted topic and a gold topic name are a valid measure of topic model utility.
- domain assumption The 27 manually recoded gold topics from Goodreads, SparkNotes, and LitCharts are a fair, unbiased ground truth.
Cite this review
Pith. "Pith review of Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes." pith.science (2026). https://pith.science/paper/6JWQHG6P
@misc{pith2026250523166,
author = {Pith},
title = {Pith review of: Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes},
year = {2026},
howpublished = {\url{https://pith.science/paper/6JWQHG6P}},
note = {Machine review of arXiv:2505.23166}
}
read the original abstract
Conventional bag-of-words approaches for topic modeling, like latent Dirichlet allocation (LDA), struggle with literary text. Literature challenges lexical methods because narrative language focuses on immersive sensory details instead of abstractive description or exposition: writers are advised to "show, don't tell." We propose Retell, a simple, accessible topic modeling approach for literature. Here, we prompt resource-efficient, generative language models (LMs) to tell what passages show, thereby translating narratives' surface forms into higher-level concepts and themes. By running LDA on LMs' retellings of passages, we can obtain more precise and informative topics than by running LDA alone or by directly asking LMs to list topics. To investigate the potential of our method for cultural analytics, we compare our method's outputs to expert-guided annotations in a case study on racial/cultural identity in high school English language arts books.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
If there are already a duplicate or relevant topic in the provided set of topics, output that topic and stop here
-
[3]
[Topic label]: [your reasoning]
Otherwise, add a new topic to the set of topics. [Document] Document Your response should be one line and ONLY contain a topic, written in the format "[Topic label]: [your reasoning]". Your response: Figure 5: Prompt for generating a possible topic pool over 1k passages. We truncate the few-shot examples here for brevity (with “...”), but show the beginni...
-
[5]
Topic modeling in embedding spaces.Trans- actions of the Association for Computational Linguis- tics, 8:439–453. Matt Erlin. 2017. Topic Modeling, Epistemology, and the English and German Novel.Journal of Cultural Analytics, 2(2). Alexander R. Fabbri, Wojciech Kry´sci´nski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval:...
work page 2017
-
[8]
KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning
Tired of topic models? Clusters of pretrained word embeddings make for fast and good topics too! InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728–1736, Online. Association for Computa- tional Linguistics. Oleg Sobchuk and Artjoms Šel,a. 2024. Computational thematics: comparing algorithms for clus...
work page Pith review arXiv 2020
-
[9]
You are a helpful assistant; follow the instructions in the prompt
Benchmarking large language models for news summarization.Transactions of the Association for Computational Linguistics, 12:39–57. Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. 2022. Describing differences between text distributions with natural language. InInter- national Conference on Machine Learning, pages 27099–27116. PMLR. Wanrong Zhu...
work page 2024
-
[13]
If there are already duplicates or relevant topics in the provided set of topics, output those topics and stop here
-
[14]
If the document contains no topic, return "None"
-
[15]
[Topic label]: [your reasoning]
Otherwise, output new, additional topic(s). [Document] Document Your response should ONLY contain topics, written in the format "[Topic label]: [your reasoning]". Your response: Figure 6: A modified version of Figure 5 for generating a possible topic pool over 1k passages, where instead of asking an LM to generate one topic per passage, it can generate mu...
Show all 17 references
-
[16]
You MUST NOT make up new topics
Topic labels must be present in the provided set of topics. You MUST NOT make up new topics
-
[17]
Output topic(s) you assign in order of their prominence in the document, with the most prominent topic first
-
[18]
[Topic label]: [your reasoning]
Each line of your response should contain a topic written in the format "[Topic label]: [your reasoning]". [Document] Document Your response should ONLY contain topics. Double check that your assignment exists in the provided set of topics! Your response: Figure 7: Prompt for ...
-
[1592]
David Bamman
PMID: 20876878. David Bamman. 2021. BookNLP. A natural language processing pipeline for books. Richard Beach, Anthony Johnston, and Amanda Haertling Thein. 2015.Identity-focused ELA teaching: A curriculum framework for diverse learners and contexts. Routledge. Melissa J. Bedfo...
2021
-
[2003]
post-racial
Latent dirichlet allocation.Journal of machine Learning research, 3(Jan):993–1022. Eduardo Bonilla-Silva. 2015. The structure of racism in color-blind, “post-racial” america.American Be- havioral Scientist, 59(11):1358–1376. Jordan Boyd-Graber, Yuening Hu, David Mimno, and 1 o...
2015 arXiv
-
[2017]
InProceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada
An automatic approach for document-level topic model evaluation. InProceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada. Association for Computational Linguistics. Federico Bianchi, Silvia Terragni, and Di...
2017
-
[2020]
Alexandra Alter
Representing race and ethnicity in american fiction, 1789-1920.Journal of Cultural Analytics, 5(2). Alexandra Alter. 2024. Book bans continue to surge in public schools. Evan P. Apfelbaum, Kristin Pauker, Samuel R. Som- mers, and Nalini Ambady. 2010. In blind pursuit of racial...
1920
-
[2024]
InFirst Conference on Language Modeling
Description-based text similarity. InFirst Conference on Language Modeling. Lisa Rhody. 2012. Topic model data for topic modeling and figurative language.Journal of Digital Humani- ties, 2. Margaret E Roberts, Brandon M Stewart, Dustin Tin- gley, Edoardo M Airoldi, and 1 other...
2012
-
[2025]
Jan Luhmann and Manuel Burghardt
Racial and Ethnic Representation in Litera- ture Taught in US High Schools.Journal of Cultural Analytics, 10(1). Jan Luhmann and Manuel Burghardt. 2022. Digital hu- manities—a discipline in its own right? An analysis of the role and position of digital humanities in the academ...
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.