Pith. sign in

REVIEW 4 major objections 4 minor 17 references

Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Running LDA on language models' abstractive retellings of literary passages yields more precise and informative topics than running LDA on the original text or asking the models to list topics directly.

desk verdict Useful, honest method paper on LM retellings for literary topic modeling, but the study-guide contamination worry is real and should be addressed. read the letter →

arxiv 2505.23166 v1 pith:6JWQHG6P submitted 2025-05-29 cs.CL

classification cs.CL
keywords topicmodelingabstractiveretellingliteraryanalysislanguagemodelslatentDirichletallocationculturalanalyticsracialidentitydistantreading
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Literary passages 'show' rather than 'tell': they express themes through sensory details, dialogue, and action, so bag-of-words topic models like LDA see surface words ('black', 'hair', 'dress') instead of the underlying ideas ('race', 'identity', 'representation'). This paper proposes Retell, which prompts a small, resource-efficient language model to retell each passage in abstractive terms, then runs LDA on those retellings. Across reader-tagged and study-guide-themed passages, Retell's most prominent topics are rated more related to the gold themes than topics from LDA alone or from directly asking language models for topic labels. In a case study on racial and cultural identity in high-school English books, Retell's topics align with expert annotations and recover more passages of interest than the baselines, suggesting a low-cost path for cultural analytics at scale.

What carries the argument

The machinery is abstractive retelling: a single prompt tells a language model to 'describe', 'summarize', or 'paraphrase' a 250-token book passage 'for a literary scholar analyzing narrative content,' and LDA is fit to the resulting one-paragraph retellings after light preprocessing (lowercasing, removing character names, filtering frequent and rare words). The prompt turns 'showing' into 'telling'—low-level sensory detail into high-level conceptual exposition—which is exactly the representation LDA's bag-of-words assumptions need. The verb choice is the control: describe and summarize encourage abstraction and outperform paraphrase, which stays closer to the surface form.

What would settle it

Build a labeled dataset in which each passage's surrounding paragraphs are manually checked to share the theme of the quote's tag, then rerun the comparison of Retell, default LDA, and TopicGPT-lite; if Retell's advantage shrinks or disappears on this clean set, the reported gains are an artifact of label-context alignment.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bottleneck in topic modeling literature is not the topic model but the representation it sees: narrative prose encodes themes implicitly, and LDA's bag-of-words assumptions fail on that surface form. Retell removes the bottleneck by having an instruction-tuned language model 'tell' what a passage 'shows'—with describe, summarize, or paraphrase prompts—and then fitting LDA to those retellings. Across four small LMs, Retell's most prominent topics are judged by crowdworkers to be 'very related' to gold themes far more often than default LDA's topics are (averaging 0.59–0.60 across models versus much lower baselines), and more informative than TopicGPT-lite's often over-broad labels such as 'human nature' or 'life'. In a case study on racial/cultural identity in U.S. high-school English books, the probability of Retell's race-related topics rises for passages that human annotators coded as mentioning or discussing race, and combining two relevant topics recovers more such passages than default LDA's single topic without losing precision.

Load-bearing premise

The evaluation assumes that the paragraph context preceding a reader-tagged or study-guide-tagged quote is about the same theme as the tag attached to the quote; if the surrounding passage is about something else, the gold labels used to score every method are unreliable and all comparisons inherit that noise.

Editorial extensions

If this is right

  • Scholars working with literary corpora can surface cross-cutting themes with a single LM prompt followed by LDA, without large API budgets.
  • Retell topics are more interpretable because they use abstractive terms (e.g., 'financial', 'racial') rather than surface words (e.g., 'money', 'black'), making them easier to relate to human notions of theme.
  • The method can screen collections for passages of interest: in the race case study, combining two relevant topics recalled more expert-annotated passages than default LDA without sacrificing precision.
  • Because topic granularity k is set after retellings are produced, researchers can adjust k without rerunning the LM.
  • Direct LM topic labeling with small models tends to collapse into a few overly broad labels; Retell avoids this by keeping LDA's distributional structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gain comes from abstraction, Retell could be combined with other lexical or embedding methods (e.g., clustering or keyword extraction) beyond LDA, though the paper only tests LDA.
  • The error analysis hints that retellings can leak book-level knowledge from pretraining, so Retell may work better on canonical or widely discussed books than on obscure ones; a comparison across familiarity would test this.
  • The method's premise suggests it will help most in genres dense with sensory detail (literary fiction) and less in expository prose, where paraphrase adds little abstraction; a genre-by-genre evaluation would quantify this.
  • For cultural analytics, Retell's topic probabilities could serve as a preliminary screening filter to direct close reading toward candidate passages, reducing the 'needle-in-haystack' annotation burden the authors describe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Retell, a topic modeling pipeline that applies LDA to large language models' abstractive retellings of literary passages rather than to the original text. It compares Retell across four LMs and three retelling verbs against two baselines: LDA on the original passages and TopicGPT-lite, a simplified version of TopicGPT that directly elicits topic labels. Evaluation proceeds through crowdworker ratings of whether a method's most prominent topic is related to scraped literary-theme labels, a passage-level topic relevance/intrusion study, and a case study on racial/cultural identity passages in books taught in U.S. high schools. The paper reports that Retell outperforms both baselines on theme relatedness and identifies more race-relevant passages in the case study, and it releases code.

Significance. If the result holds, Retell offers a low-cost, easy-to-use method for humanities researchers who want to surface cross-cutting themes from literary text with smaller LMs. The paper is strong on transparency: it releases code, tests multiple models and retelling verbs, includes human annotation in two evaluation modes, and accompanies the case study with a detailed codebook and explicit limitations. Its main weakness is that the empirical evidence for the stated mechanism is undercut by the lack of any control for pretraining contamination from the very study-guide sources used as gold labels, and by the absence of uncertainty measures in the headline evaluation. The method is promising but the central claim, as currently worded, is broader than the evidence supports.

major comments (4)
  1. [§4.1, §6.2.2] The evaluation's gold labels are scraped from Goodreads, SparkNotes, and LitCharts (§4.1), all of which are plausible components of the pretraining data for the tested LMs. The paper provides no control for this overlap, so the observed advantage of Retell over default LDA and TopicGPT-lite could arise from the LM reproducing memorized study-guide themes rather than from abstractive retelling of the passage. The paper's own error analysis in §6.2.2 documents exactly this mechanism: a retelling includes a character's Iranian background that is absent from the passage, which the authors attribute to the LM's familiarity with the book. This is load-bearing because the central claim is about the method's ability to translate surface form into higher-level concepts; without a control using post-cutoff or obscure texts, or a test that removes identifying book-level information, the mechanism and the method's generality to unpublished or niche works remain unverified.
  2. [Table 2, Table 8] The headline result in Table 2 (and the extended version in Table 8) is reported as rating proportions without any uncertainty quantification. No confidence intervals, standard errors, or significance tests are given for the comparisons across methods or across values of k. This makes the central claim that Retell consistently outperforms baselines difficult to evaluate, especially since some differences (e.g., certain Retell variants versus TopicGPT-lite) are numerically small. The authors should report bootstrap confidence intervals over the 60 passage sets and appropriate paired tests, and apply the same rigor to the precision/recall comparisons in Appendix C.1.
  3. [Abstract, §5, Appendix C.4] The abstract and Section 5 state the central claim without the resource-efficiency qualifier that the evidence actually supports. Appendix C.4 shows that with GPT-4o, TopicGPT-lite achieves similar crowd ratings to Retell-describe and higher precision and recall (Table 10). The conclusion's phrasing 'especially for resource-efficient, small LMs' is accurate, but the abstract and Section 5 should be revised so that the general claim is scoped to smaller LMs, or additional evidence should be provided for larger LMs.
  4. [§4.1] The evaluation assumes that the paragraph context preceding a Goodreads, SparkNotes, or LitCharts quote does not deviate far from the tag or theme attached to the quote. This assumption is stated but not validated, despite a quote-matching rate of only 68.0 and the inclusion of up to 250 tokens of preceding context. If the surrounding passage differs thematically from the quote, the gold labels are noisy for every method, and all comparisons in Table 2 inherit that noise. The authors should either validate the assumption on a sample (e.g., human ratings of whether the full passage matches the quote's theme) or restrict evaluation to passages where the quote is representative of the whole window. The manual recoding of scraped tags/themes into 27 topics also lacks inter-annotator reliability, which is important because this recoding defines the gold set.
minor comments (4)
  1. [Table 2] The column structure of Table 2 is difficult to parse; the 'Rating /k' row lists many numbers without visually separating model/k combinations, and the repeated '✓', '?', and '✗' rows make it hard to map ratings to methods. A grouped layout or a separate table per model would improve readability.
  2. [Table 3] The note that ratings 'differ significantly for each method' is ambiguous; please specify which pairwise comparisons are significant (e.g., Topic 1 versus Intruder) and whether any multiple-comparison correction was applied.
  3. [§5] The phrase 'different instructive verbs' appears to be a slip for 'retelling verbs' or 'instruction verbs'; the current wording is imprecise.
  4. [Appendix C.1, Table 9] The absolute precision values in Table 9 are low (e.g., 0.128 for Retell-summarize at k=50); the authors should report the base rate of positive passage pairs so that readers can judge the effect size, and should comment on whether precision at these levels is practically useful for exploratory literary analysis.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Retell is an unsupervised pipeline evaluated against external gold labels; the minor self-citations provide data or error-analysis context but do not carry the central claim.

full rationale

The paper's central claim is an empirical comparison: LDA is fit to language-model retellings of passages using default decoding parameters, with no parameter tuned to the gold labels or human ratings (§3.1, §5). The pipeline is therefore not self-definitional or a fitted-input-called-prediction case: the retellings are generated from the passage alone, and the evaluation labels come from external sources (§4.1). The few self-citations (Lucy et al. 2025 for the ELA booklist, Levine et al. 2021 for teacher-listed books, and Chang et al. 2023 in the error analysis) supply data or an explanation for memorized book context, but the main topic-modeling comparison does not reduce to those citations. The Limitations section explicitly acknowledges that LMs can reproduce book-level knowledge from pretraining and that online gold labels carry content-production bias; these are real validity threats, but they are not circular reductions, because the method's output is not defined in terms of the gold labels and no fitted parameter is renamed as a prediction. No circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method itself has no free parameters fitted to the gold labels: k and the 250-token passage window are experimental choices, not estimates. The evaluation rests on the domain assumptions listed, especially that preceding context shares a quote's theme and that LM retellings preserve thematic content. No invented entities are introduced.

free parameters (3)
  • LDA topic count k = 50 (also 100, 200, or TopicGPT-determined values 14, 89, 157, 206, 538)
    Topic granularity is set by the experimenter, not fitted to gold labels; results vary somewhat with k.
  • Passage context window = 250 whitespace tokens
    Passages are capped at 250 tokens; the surrounding context of a gold-labeled quote is assumed to carry the same theme (Section 4.1).
  • Case-study keyword sampling threshold = PMI >= 0.15
    Passages for annotation are sampled only if they contain seed keywords, with a PMI threshold used to expand the seed list (Appendix D.1); this conditions the case study on keyword presence.
assumptions (4)
  • domain assumption The paragraph preceding a quoted theme label has the same theme as the label itself.
    Stated in Section 4.1 as a made assumption used to construct 11.6k labeled passages; if false, gold labels are noisy.
  • domain assumption LM retellings are faithful enough that LDA topics over them preserve, rather than distort, the thematic content of passages.
    The entire method depends on retellings surfacing implicit content; the paper's own error analysis documents cases where retellings add book-level context or omit racist content (Section 6.2.2).
  • domain assumption Crowdworker ratings of semantic relatedness between a predicted topic and a gold topic name are a valid measure of topic model utility.
    Used for the headline comparison in Table 2; no alternative objective metric is primary.
  • domain assumption The 27 manually recoded gold topics from Goodreads, SparkNotes, and LitCharts are a fair, unbiased ground truth.
    Authors manually sorted and recoded scraped tags and themes; acknowledged as potentially biased by creator population (Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes." pith.science (2026). https://pith.science/paper/6JWQHG6P

@misc{pith2026250523166,
  author       = {Pith},
  title        = {Pith review of: Tell, Don't Show: Leveraging Language Models' Abstractive Retellings to Model Literary Themes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JWQHG6P}},
  note         = {Machine review of arXiv:2505.23166}
}
read the original abstract

Conventional bag-of-words approaches for topic modeling, like latent Dirichlet allocation (LDA), struggle with literary text. Literature challenges lexical methods because narrative language focuses on immersive sensory details instead of abstractive description or exposition: writers are advised to "show, don't tell." We propose Retell, a simple, accessible topic modeling approach for literature. Here, we prompt resource-efficient, generative language models (LMs) to tell what passages show, thereby translating narratives' surface forms into higher-level concepts and themes. By running LDA on LMs' retellings of passages, we can obtain more precise and informative topics than by running LDA alone or by directly asking LMs to list topics. To investigate the potential of our method for cultural analytics, we compare our method's outputs to expert-guided annotations in a case study on racial/cultural identity in high school English language arts books.

Figures

Figures reproduced from arXiv: 2505.23166 by the authors.

Figure 1
Figure 1. Language models (LMs) can support tradi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Precision and recall (y-axes) of using relevant Retell and LDA topics’ probability thresholds (x-axes) to distinguish different types of human-labeled passages from each other (discuss, mention, neither). Examining against compares only one topic per method, and is more precise. Comparing with considers all relevant topics each method surfaced, and has higher recall without sacrificing precision. Left two plots: ide… view at source ↗
Figure 3
Figure 3. Additional evaluation results for k = 50, 100, and 200, disaggregated by the source of ground truth labels. Dashed gray lines indicate default LDA’s perfor￾mance, and the models on the x-axis include Llama 3.1 8B (L), Phi-3.5-mini (P), Gemma 2 2B (G) and GPT-4o mini (O). Since TopicGPT-lite is not designed for use with predetermined k, it is not included in these plots [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Results for two different runs of Llama 3.1 8B [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Prompt for generating a possible topic pool over 1k passages. We truncate the few-shot examples here for [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: A modified version of Figure [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Prompt for assigning topics to passages. Few-shot examples are shortened with “...” for brevity purposes, [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages

  1. [1]

    If there are already a duplicate or relevant topic in the provided set of topics, output that topic and stop here

  2. [3]

    [Topic label]: [your reasoning]

    Otherwise, add a new topic to the set of topics. [Document] Document Your response should be one line and ONLY contain a topic, written in the format "[Topic label]: [your reasoning]". Your response: Figure 5: Prompt for generating a possible topic pool over 1k passages. We truncate the few-shot examples here for brevity (with “...”), but show the beginni...

  3. [5]

    Matt Erlin

    Topic modeling in embedding spaces.Trans- actions of the Association for Computational Linguis- tics, 8:439–453. Matt Erlin. 2017. Topic Modeling, Epistemology, and the English and German Novel.Journal of Cultural Analytics, 2(2). Alexander R. Fabbri, Wojciech Kry´sci´nski, Bryan Mc- Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval:...

  4. [8]

    KRISTEVA: Close Reading as a Novel Task for Benchmarking Interpretive Reasoning

    Tired of topic models? Clusters of pretrained word embeddings make for fast and good topics too! InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1728–1736, Online. Association for Computa- tional Linguistics. Oleg Sobchuk and Artjoms Šel,a. 2024. Computational thematics: comparing algorithms for clus...

  5. [9]

    You are a helpful assistant; follow the instructions in the prompt

    Benchmarking large language models for news summarization.Transactions of the Association for Computational Linguistics, 12:39–57. Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. 2022. Describing differences between text distributions with natural language. InInter- national Conference on Machine Learning, pages 27099–27116. PMLR. Wanrong Zhu...

  6. [13]

    If there are already duplicates or relevant topics in the provided set of topics, output those topics and stop here

  7. [14]

    If the document contains no topic, return "None"

  8. [15]

    [Topic label]: [your reasoning]

    Otherwise, output new, additional topic(s). [Document] Document Your response should ONLY contain topics, written in the format "[Topic label]: [your reasoning]". Your response: Figure 6: A modified version of Figure 5 for generating a possible topic pool over 1k passages, where instead of asking an LM to generate one topic per passage, it can generate mu...

Show all 17 references
  1. [16]

    You MUST NOT make up new topics

    Topic labels must be present in the provided set of topics. You MUST NOT make up new topics

  2. [17]

    Output topic(s) you assign in order of their prominence in the document, with the most prominent topic first

  3. [18]

    [Topic label]: [your reasoning]

    Each line of your response should contain a topic written in the format "[Topic label]: [your reasoning]". [Document] Document Your response should ONLY contain topics. Double check that your assignment exists in the provided set of topics! Your response: Figure 7: Prompt for ...

  4. [1592]

    David Bamman

    PMID: 20876878. David Bamman. 2021. BookNLP. A natural language processing pipeline for books. Richard Beach, Anthony Johnston, and Amanda Haertling Thein. 2015.Identity-focused ELA teaching: A curriculum framework for diverse learners and contexts. Routledge. Melissa J. Bedfo...

  5. [2003]

    post-racial

    Latent dirichlet allocation.Journal of machine Learning research, 3(Jan):993–1022. Eduardo Bonilla-Silva. 2015. The structure of racism in color-blind, “post-racial” america.American Be- havioral Scientist, 59(11):1358–1376. Jordan Boyd-Graber, Yuening Hu, David Mimno, and 1 o...

  6. [2017]

    InProceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada

    An automatic approach for document-level topic model evaluation. InProceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 206–215, Vancouver, Canada. Association for Computational Linguistics. Federico Bianchi, Silvia Terragni, and Di...

  7. [2020]

    Alexandra Alter

    Representing race and ethnicity in american fiction, 1789-1920.Journal of Cultural Analytics, 5(2). Alexandra Alter. 2024. Book bans continue to surge in public schools. Evan P. Apfelbaum, Kristin Pauker, Samuel R. Som- mers, and Nalini Ambady. 2010. In blind pursuit of racial...

  8. [2024]

    InFirst Conference on Language Modeling

    Description-based text similarity. InFirst Conference on Language Modeling. Lisa Rhody. 2012. Topic model data for topic modeling and figurative language.Journal of Digital Humani- ties, 2. Margaret E Roberts, Brandon M Stewart, Dustin Tin- gley, Edoardo M Airoldi, and 1 other...

  9. [2025]

    Jan Luhmann and Manuel Burghardt

    Racial and Ethnic Representation in Litera- ture Taught in US High Schools.Journal of Cultural Analytics, 10(1). Jan Luhmann and Manuel Burghardt. 2022. Digital hu- manities—a discipline in its own right? An analysis of the role and position of digital humanities in the academ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.