Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Agentic Retrieval of Topics and Insights from Earnings Calls

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that an LLM-driven agentic pipeline—topic retriever plus ontologist—can extract financially relevant topics from earnings calls and organize them into a hierarchical ontology that stays current as new…

desk verdict A plausible agentic topic-ontology pipeline for earnings calls, but the evaluation rests on a single run with an unspecified matching threshold; the paper is honest about limits and deserves revision, not rejection. read the letter →

arxiv 2507.07906 v1 pith:VAQRREYP submitted 2025-07-10 cs.LG cs.AI

Anant Gupta , Rajarshi Bhowmik , Geoffrey Gunow This is my paper
classification cs.LGcs.AI
keywords TopicsOntologyInsightsRetrievalFinancialdocumentsEarningsCallsLLMagentsTrendanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Earnings calls are dense sources of strategic information, but traditional topic models need a pre-vetted list of topics and often miss emerging themes. The paper proposes an LLM-agent framework that extracts topics directly from each paragraph, checks whether a topic already exists via semantic equivalence, and if not, inserts it into a tree-structured ontology under the most specific matching parent. This lets the topic hierarchy grow with the data instead of being fixed in advance. The authors show the constructed ontology is internally coherent, that mention-frequency trends align with known industry shifts, and that the framework surfaces emerging topics in EV and semiconductor sectors that analysts would want to track. The value, if sustained, is a scalable way to monitor strategic narratives across companies and quarters without manual topic curation.

What carries the argument

The central mechanism is the semantic-equivalence check performed by the Ontologist agent before any topic is added. The agent is prompted to determine whether a candidate topic matches an existing node or alias with bidirectional equivalence—same scope and specificity—rather than treating it as a parent or subset category. This single gate controls the entire ontology: it decides whether new topics are named consistently, whether aliases accrue, and whether parent-child relations stay at comparable granularity. Supporting it are the Topic Retriever's paragraph-level extraction prompt and a tree ontology where each node stores a UUID, a name, aliases, and creation/update timestamps. The coherence evaluation, which measures embedding similarity between parent and child nodes against random pairings, provides the paper's quantitative check on the whole construction.

What would settle it

Take a random sample of, say, 500 paragraphs from the 141 transcripts, and have two financial analysts independently assign canonical topic labels and parent categories to each excerpt. If the LLM-retrieved topics disagree with the analysts on more than a small fraction of paragraphs—or if the parent-child edges in the ontology are frequently judged wrong—then the ontology's coherence score and the downstream trend and competitor analyses lose their foundation.

Watch

Extended reading notes

Core claim

On its own terms, this paper establishes that a two-agent LLM pipeline can discover and maintain a hierarchical topic ontology from earnings-call transcripts. A Topic Retriever reads paragraph-level text and outputs financially relevant topic names with short excerpts; an Ontologist agent then decides, by semantic equivalence, whether a retrieved topic is already present—and if not, determines the narrowest parent node under which to insert it. Running this over 141 transcripts from 12 companies across EV and semiconductor sectors yields an ontology of 3,200 nodes with a median of 2.5 aliases per node and a four-level depth. The paper validates the ontology by showing that parent-child pairs have substantially higher embedding similarity than random pairings, and it demonstrates downstream utility by identifying statistically significant rising and falling topics (e.g., 'Supply Chain' declining, 'Generative AI' rising) and by comparing companies via Jaccard similarity over their top-100 topics.

Load-bearing premise

The load-bearing premise is that the LLM's topic extraction and semantic-matching decisions are reliable enough that the ontology stays coherent and the resulting mention counts mean what they appear to mean; the paper acknowledges it has no gold-standard annotations to verify this.

Editorial extensions

If this is right

  • Financial analysts can track how often and how directionally topics are mentioned quarter by quarter, spotting trends like the decline of 'Supply Chain' discussions as soon as the call is released.
  • The hierarchical ontology with aliases lets retrieval recognize synonymous topic names ('M&A' and 'Mergers & Acquisitions'), so downstream queries on a theme aggregate mentions that would otherwise be scattered.
  • The same pipeline can be pointed at new sectors or document collections without retraining a supervised topic model, as long as the LLM prompt generalizes.
  • Comparing competitors by shared top topics (Jaccard similarity over leaf nodes) gives a qualitative complement to financial ratios, exposing strategic focus as well as performance.
  • If the ontology stays coherent as it grows, emerging-topic lists such as 'on-premises AI,' 'AI copilots,' and 'low-cost vehicles' can feed early signal for equity research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation statement admits there is no gold-standard annotation for topics; a direct implication is that the ontology's reliability should be measured against expert labels on a sample of paragraphs before production use in high-stakes financial decisions.
  • Because the entire analysis rests on the LLM's topic naming and matching, a cheaper or more deterministic alternative (e.g., a small fine-tuned classifier for existence checks) could achieve the same ontology-growth behaviour with less cost and variance; the paper does not test this.
  • The trend analysis uses Kendall's tau on raw mention counts; a natural extension would be to test whether the same significance holds when counts are normalized by call length or when model-blinded to the sector, to rule out confounds like call duration.
  • If the framework is correct, a strict out-of-time test—building the ontology on 2021–2023 calls and asking whether it surfaces topics that 2024 calls actually discuss—would directly validate its emerging-topic claims; the paper only shows post-hoc identified trends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an LLM-agent framework that extracts topics from earnings call transcripts, organizes them into a hierarchical ontology with parent-child relationships, and uses the ontology for trend analysis, competitor analysis, and emerging-topic detection. The pipeline consists of a Topic Retriever, an Ontology data structure, and an Ontologist agent that checks semantic equivalence and inserts new topics. The system is demonstrated on 141 earnings call transcripts from 12 companies in the EV and semiconductor sectors over 2021-2024. Evaluation includes a qualitative comparison with LDA, an embedding-based coherence check of the ontology, and qualitative validation of selected trends against public reports. The authors explicitly acknowledge the absence of gold-standard annotations and the presence of noisy topics in the Limitations section.

Significance. If the framework were shown to be stable, reproducible, and aligned with expert judgment, it would provide a valuable tool for financial analysts, enabling timely discovery of emerging themes without a pre-vetted topic list. The paper's strengths include a concrete dataset, full prompts in the appendix, and a clearly described pipeline; the ontology coherence check is a reasonable internal sanity test. However, the evidence presented mostly establishes internal consistency rather than external validity: the central claims about emerging topics and trends are not validated against a ground truth, and the stochastic nature of the pipeline is not characterized. The contribution is therefore suggestive rather than demonstrated.

major comments (3)
  1. [Section 3.4.1, Appendix A.3, Table 5] The semantic matching step that decides whether a newly extracted topic already exists in the ontology is load-bearing for every downstream number, but the paper never specifies the acceptance threshold for the similarity scores returned by the prompt in Appendix A.3, nor does it report the LLM version, temperature, or random seed. The ontology statistics in Table 5 (3,200 nodes, mean aliases per node 2.5 ± 5.56) are therefore artifacts of an undocumented decision rule, and the trend statistics in Section 5.1 and emerging-topic lists in Section 5.3 are computed from a single stochastic run. Without a sensitivity analysis on the match threshold or repeated runs with reported variance, the claim that the framework 'dynamically captures emerging topics' is not yet supported.
  2. [Section 4.2.1, Table 8, Section 6] The ontology coherence evaluation measures cosine similarity between parent and child topics that the same LLM pipeline generated, compared with random parent-child pairings. This demonstrates internal consistency but not correspondence to any external truth; the Limitations section explicitly concedes a 'Lack of Gold-Standard Annotations' and 'Noisy Topics.' The trend and emerging-topic validations in Sections 5.1 and 5.3 are post-hoc selections of examples that align with public reports, not pre-registered predictions or quantitative comparisons with a ground truth. The central claim of discovering emerging topics therefore requires expert-labeled or independently derived topic labels to be convincing.
  3. [Section 5.1, Tables 9 and 10] Trend significance is assessed with Kendall's tau on mention frequencies computed from the single ontology produced by the unvalidated matcher. The paper reports no confidence intervals, repeated stochastic runs, or robustness checks across ontology construction parameters, so the significance statements in Tables 9 and 10 are conditional on an unverified intermediate output. At minimum, the authors should report the variability of trend classifications across repeated runs and across the unspecified match threshold.
minor comments (5)
  1. [Table 9] The word 'restructring' should be 'restructuring.'
  2. [Section 2] The text 'We chooseEVsandSemiconductorsas our sectors' contains a typo: 'EVs' is rendered as 'EVSs.'
  3. [Tables 6 and 7] The entry '1.000.25' appears to be a formatting artifact; it should read '1.00 0.25' (and similarly in Table 7).
  4. [Section 4.2] The 32 root and 361 second-level seed topics are introduced without provenance; a brief description of how they were chosen and whether the results are robust to this choice would help.
  5. [Section 5.1] The claim that insights 'appear significantly earlier' than financial media is reported without quantitative timestamps or a systematic comparison; this should be presented as anecdotal or supported with dates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system is an empirical pipeline whose outputs (ontology, trends, emerging topics) are not definitionally equivalent to its inputs.

full rationale

The paper does not make a first-principles derivation or a fitted-parameter prediction; it builds an LLM-driven extraction and ontology-construction pipeline and then reports descriptive downstream statistics. The ontology coherence evaluation (Section 4.2.1) uses a separate embedding model (all-MiniLM-L6-v2) to compare actual parent-child pairs against random pairings, which is an internal-consistency check rather than a claim that a quantity is predicted from its own definition. Topic-existence matching (Section 3.4.1) and trend/emerging-topic analyses (Sections 5.1 and 5.3) are empirical computations whose values depend on the LLM and an unspecified similarity threshold, but no equation reduces the reported trends to the matcher's inputs by construction. The acknowledged limitations (lack of gold-standard annotations, noisy topics, subjectivity) and the single-run, unspecified-threshold reproducibility issue are validity and robustness concerns, not circularity. There are no load-bearing self-citations and no imported uniqueness theorem. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central method has no fitted constants, but it depends on hand-selected seed topics and undisclosed LLM settings. The main assumptions are about the reliability of LLM extraction and the representativeness of paragraph segmentation. No new physical entities are introduced.

free parameters (2)
  • Seed topic ontology size = 32 root topics, 361 first-level topics
    Hand-specified initial ontology; all later insertions and coherence statistics depend on this starting point.
  • LLM configuration = not disclosed
    Model choice, temperature, and decoding settings affect extraction and ontology decisions but are not reported.
assumptions (3)
  • domain assumption Paragraphs are coherent units for topic extraction
    The pipeline segments transcripts by paragraph and extracts topics independently per paragraph (Section 3.2), which assumes each paragraph contains self-contained topical content.
  • domain assumption LLM semantic judgments are sufficiently accurate for ontology construction
    The Ontologist relies on LLM judgments for semantic equivalence and parent selection (Sections 3.4.1 and 3.4.2); no gold standard validates these judgments.
  • ad hoc to paper The fixed seed ontology provides a neutral and adequate starting point
    An ontology initialized with 32 root and 361 first-level topics (Section 4.2) shapes all subsequent insertions; this seed set is chosen by the authors rather than derived from data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agentic Retrieval of Topics and Insights from Earnings Calls." pith.science (2026). https://pith.science/paper/VAQRREYP

@misc{pith2026250707906,
  author       = {Pith},
  title        = {Pith review of: Agentic Retrieval of Topics and Insights from Earnings Calls},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VAQRREYP}},
  note         = {Machine review of arXiv:2507.07906}
}
read the original abstract

Tracking the strategic focus of companies through topics in their earnings calls is a key task in financial analysis. However, as industries evolve, traditional topic modeling techniques struggle to dynamically capture emerging topics and their relationships. In this work, we propose an LLM-agent driven approach to discover and retrieve emerging topics from quarterly earnings calls. We propose an LLM-agent to extract topics from documents, structure them into a hierarchical ontology, and establish relationships between new and existing topics through a topic ontology. We demonstrate the use of extracted topics to infer company-level insights and emerging trends over time. We evaluate our approach by measuring ontology coherence, topic evolution accuracy, and its ability to surface emerging financial trends.

Figures

Figures reproduced from arXiv: 2507.07906 by the authors.

Figure 1
Figure 1. Distribution of paragraph lengths (defined by num [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. for an overview of the system. Initially, the topic retriever analyzes paragraph content to iden￾tify and extract financially-relevant topics along with correspond￾ing textual excerpts. The ontologist then assesses each identified topic’s novelty by verifying its presence within the existing ontol￾ogy, systematically adding new topics or updating existing ones to reflect ongoing narrative developments. The ontology … view at source ↗
Figure 3
Figure 3. Count of topics discovered over time as we only [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Trend of "Supply Chain" in semiconductor industry. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Trend of "Supply Chain" in EV industry [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Trending up topics from NVDA (left) and LCID (right). [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: 15 topics discovered by LDA baseline method represented by 10 keywords each where the color represents the u_mass [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

  1. [1]

    Managers set the tone: Equity incentives and the tone of earnings press releases.Journal of Banking & Finance, 72:S132–S147, 2016

    Özgür Arslan-Ayaydin, Kris Boudt, and James Thewissen. Managers set the tone: Equity incentives and the tone of earnings press releases.Journal of Banking & Finance, 72:S132–S147, 2016

    work page 2016

  2. [2]

    Evaluating Large Language Models on Financial Report Summarization: An Empirical Study

    Xinqi Yang, Scott Zang, Yong Ren, Dingjie Peng, and Zheng Wen. Evaluating large language models on financial report summarization: An empirical study. arXiv preprint arXiv:2411.06852, 2024

    work page Pith review arXiv 2024

  3. [3]

    Evaluating sentiment in annual reports for financial distress prediction using neural networks and support vector machines

    Petr Hájek and Vladimír Olej. Evaluating sentiment in annual reports for financial distress prediction using neural networks and support vector machines. In Engineering Applications of Neural Networks: 14th International Conference, EANN 2013, Halkidiki, Greece, September 13-16, 2013 Proceedings, Part II 14, pages 1–10. Springer, 2013

    work page 2013

  4. [4]

    Finbert: Financial sentiment analysis with pre-trained language models.arXiv preprint arXiv:1908.10063, 2019

    Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models.arXiv preprint arXiv:1908.10063, 2019. FinIR ’25, July 17, 2025, Padua, Italy Gupta et al. AMD (19) INTC (12) ASML (8) A VGO (16) QCOM (8) AI inference, AI infrastructure, AI strategy, brand partnerships, buy- backs, capital expenditures, cloud partnerships, data center, ga...

    arXiv 1908

  5. [5]

    Bloomberggpt: A large language model for finance, 2023.URL https://arxiv

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023.URL https://arxiv. org/abs/2303.17564, 2024

    arXiv 2023

  6. [6]

    Topic Modelling Meets Deep Neural Networks: A Survey

    He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. Topic modelling meets deep neural networks: A survey.arXiv preprint arXiv:2103.00498, 2021

    work page Pith review arXiv 2021

  7. [7]

    Distilled ChatGPT Topic & Sentiment Modeling with Applications in Finance

    Olivier Gandouet, Mouloud Belbahri, Armelle Jezequel, and Yuriy Bodjov. Dis- tilled chatgpt topic & sentiment modeling with applications in finance.arXiv preprint arXiv:2403.02185, 2024

    work page Pith review arXiv 2024

  8. [8]

    Latent dirichlet allocation

    David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003

    2003

Show all 24 references
  1. [9]

    Large language models for scholarly ontology generation: An extensive analysis in the engineering field.arXiv preprint arXiv:2412.08258, 2024

    Tanay Aggarwal, Angelo Salatino, Francesco Osborne, and Enrico Motta. Large language models for scholarly ontology generation: An extensive analysis in the engineering field.arXiv preprint arXiv:2412.08258, 2024

    2024 arXiv

  2. [10]

    Packt Publishing Ltd, 2020

    Stefan Jansen.Machine Learning for Algorithmic Trading: Predictive models to extract signals from market and alternative data for systematic trading strategies with Python. Packt Publishing Ltd, 2020

    2020

  3. [11]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  4. [12]

    Practical guide to using kendall’s 𝜏 in the context of forecasting critical transitions.Royal Society Open Science, 9(7):211346, 2022

    Shiyang Chen, Amin Ghadami, and Bogdan I Epureanu. Practical guide to using kendall’s 𝜏 in the context of forecasting critical transitions.Royal Society Open Science, 9(7):211346, 2022

    2022

  5. [13]

    Bertrend: Neural topic modeling for emerging trends detection.arXiv preprint arXiv:2411.05930, 2024

    Allaa Boutaleb, Jerome Picault, and Guillaume Grosjean. Bertrend: Neural topic modeling for emerging trends detection.arXiv preprint arXiv:2411.05930, 2024. Agentic Retrieval of Topics and Insights from Earnings Calls FinIR ’25, July 17, 2025, Padua, Italy A Appendix A.1 LDA t...

    2024 arXiv

  6. [14]

    pharmacy benefits managers

    Extract topic from the given document text. These topics are of interest to a financial analyst. You are encouraged to tag multiple topics, both general and specific. 1.a. Some examples of general topics: guidance, capex, dividends, buybacks, M&A, macro environment, labor, sup...

  7. [15]

    topic_name

    Also extract the excerpts that mention those topics. 2.a. You may re-word the excerpt to scope it out better and provide a succinct summary of the context. 2.b. Do not speculate or provide opinions. Base all observations strictly on the information present in the documents. Th...

  8. [16]

    Topics must be semantically equivalent in scope and specificity

  9. [17]

    IMPORTANT: Matched topics must NOT be parent categories or supersets of the query topic

  10. [18]

    Brand and product design

    The relationship must be bidirectional - each topic should be able to substitute for the other Examples of Invalid Matches (Parent/Subset Relationships): Query: "Brand and product design" != "Marketing and Advertising" Reason: Marketing is a broader category that contains bran...

    2025

  11. [19]

    Analyze the core meaning and key concepts of the query topic

  12. [20]

    For each topic in the list, evaluate: - Core concepts and ideas - Subject matter and domain - Scope and specificity level - Intent and context - Parent/subset relationship check

  13. [21]

    query_topic

    For potential matches: - Verify its not a parent category or superset - Rate similarity on a scale of 0-100% - Explain your reasoning with explicit mention of scope equivalence Given a List of reference topics and a Query topic, your response should be as a JSON object as foll...

    2025

  14. [22]

    Start by finding the most appropriate super-parent topic for the given topic

  15. [23]

    Identify the most specific child topic that can be a parent of the give **topic**

  16. [24]

    Roboadvisor

    If there is no such parent present, then assign the super-parent topic as the parent topic of the given **topic** Input: - Given Topic: [Name of the topic to categorize] - Topic Tree: {Super-parent topic: [list of potential parent topics], super-parent topic 2: [list of potent...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.