{"id":"6fa44693-eaec-4a36-a6c4-5ad2b2774170","arxiv_id":"2507.07906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM agent extracts financial topics from earnings calls, builds a hierarchical topic ontology, and uses topic mention trends to flag rising and falling themes.","lead":"The paper presents an AI system that reads earnings call transcripts, extracts financially important topics, and organizes them into a growing family tree of related concepts. It is aimed at financial analysts who want to see which themes are rising or fading for a company or sector.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"A single stochastic run with an unspecified semantic-match threshold underlies all trend and emerging-topic results; without a stability or sensitivity check the central claim is not yet supported.","rationale":"I read the paper in good faith. The proposed agentic framework is clearly described, the dataset is carefully scoped, and the qualitative examples in Tables 1, 11, and 12 give a plausible illustration of what the system can produce. The LDA comparison and the embedding-coherence test are reasonable first steps, but they do not validate the temporal claims that form the abstract's central promise: discovering emerging topics and tracking trends over time.\n\nThe reader's weakest assumption, that LLM topic extraction and semantic matching are reliable enough to build the ontology, is correct. My concern sharpens that assumption into a concrete reproducibility issue. The ontology is a single stochastic output of LLM calls with no reported sampling parameters and no sensitivity analysis. The similarity threshold in the Topic Existence module is the hinge connecting extracted topic names to ontology nodes, yet it is never stated or varied. Because the trend and emerging-topic results are computed from this one ontology, they could change substantially if the pipeline were rerun. This is not a logical inconsistency in the method, but it is a correctness risk: the central claim is about a system that reliably surfaces emerging topics, and a single-run demonstration cannot support that claim.\n\nI do not see a reason to reject the paper. The idea is sound and the limitations are honestly discussed, but the evaluation is not yet sufficient. I agree with the reader's CONDITIONAL verdict and recommend no change. The concrete test I propose would settle the reproducibility question directly: if the emerging-topic lists are stable across reruns, the main concern is resolved; if not, the paper would need substantial additional evidence before the claim is credible.","tokens_in":10934,"tokens_out":6672,"duration_ms":81155,"concrete_test":"Run the full pipeline five times on the same 141 transcripts, varying only the random seed and/or LLM temperature (e.g., 0.0, 0.3, 0.7, 1.0) while keeping prompts and seed topics fixed. Rebuild the ontology each time and recompute the Section 5.3 emerging-topic lists and the trending-topic lists in Tables 9 and 10. Measure the Jaccard overlap of the top-10 emerging topics per sector and the variance in total ontology node count. If the overlap falls below 0.5 or the node count varies by more than 10%, the trending and emerging results are not reproducible and the central claim is unsupported. As a secondary check, if a numeric similarity threshold is used in Topic Existence, vary it (e.g., 0.7, 0.85, 0.95) and report the effect on ontology size and the emerging-topic lists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire ontology is built by one pass over the corpus, and every downstream number depends on the Topic Existence matcher (Section 3.4.1). The prompt in Appendix A.3 returns similarity scores, but the paper never states the threshold used to accept a match, nor the LLM version, temperature, or random seed. The final ontology size (3,200 nodes) and alias counts (2.5 ± 5.56) are therefore artifacts of an unspecified decision rule. If the threshold is too permissive, distinct concepts are merged and later mentions are aliases of old nodes, suppressing genuinely new topics; if too strict, the same concept is split into several nodes and can later appear as a spurious 'emerging' topic. Tables 9 and 10 and the Section 5.3 emerging-topic lists are computed from this single ontology, so they inherit whatever bias the matcher introduces. The only quantitative internal check, Table 8, measures embedding similarity of names and does not validate temporal mention counts or the emerging-topic definition. The Limitations section concedes noisy topics and absent gold-standard labels, but it does not address run-to-run reproducibility, which is required before 'dynamically capture emerging topics' can be accepted as a reliable system property.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-agent framework that extracts topics from earnings call transcripts, organizes them into a hierarchical ontology with parent-child relationships, and uses the ontology for trend analysis, competitor analysis, and emerging-topic detection. The pipeline consists of a Topic Retriever, an Ontology data structure, and an Ontologist agent that checks semantic equivalence and inserts new topics. The system is demonstrated on 141 earnings call transcripts from 12 companies in the EV and semiconductor sectors over 2021-2024. Evaluation includes a qualitative comparison with LDA, an embedding-based coherence check of the ontology, and qualitative validation of selected trends against public reports. The authors explicitly acknowledge the absence of gold-standard annotations and the presence of noisy topics in the Limitations section.","tokens_in":11116,"tokens_out":4575,"duration_ms":43277,"significance":"If the framework were shown to be stable, reproducible, and aligned with expert judgment, it would provide a valuable tool for financial analysts, enabling timely discovery of emerging themes without a pre-vetted topic list. The paper's strengths include a concrete dataset, full prompts in the appendix, and a clearly described pipeline; the ontology coherence check is a reasonable internal sanity test. However, the evidence presented mostly establishes internal consistency rather than external validity: the central claims about emerging topics and trends are not validated against a ground truth, and the stochastic nature of the pipeline is not characterized. The contribution is therefore suggestive rather than demonstrated.","major_comments":[{"comment":"The semantic matching step that decides whether a newly extracted topic already exists in the ontology is load-bearing for every downstream number, but the paper never specifies the acceptance threshold for the similarity scores returned by the prompt in Appendix A.3, nor does it report the LLM version, temperature, or random seed. The ontology statistics in Table 5 (3,200 nodes, mean aliases per node 2.5 ± 5.56) are therefore artifacts of an undocumented decision rule, and the trend statistics in Section 5.1 and emerging-topic lists in Section 5.3 are computed from a single stochastic run. Without a sensitivity analysis on the match threshold or repeated runs with reported variance, the claim that the framework 'dynamically captures emerging topics' is not yet supported.","section":"Section 3.4.1, Appendix A.3, Table 5"},{"comment":"The ontology coherence evaluation measures cosine similarity between parent and child topics that the same LLM pipeline generated, compared with random parent-child pairings. This demonstrates internal consistency but not correspondence to any external truth; the Limitations section explicitly concedes a 'Lack of Gold-Standard Annotations' and 'Noisy Topics.' The trend and emerging-topic validations in Sections 5.1 and 5.3 are post-hoc selections of examples that align with public reports, not pre-registered predictions or quantitative comparisons with a ground truth. The central claim of discovering emerging topics therefore requires expert-labeled or independently derived topic labels to be convincing.","section":"Section 4.2.1, Table 8, Section 6"},{"comment":"Trend significance is assessed with Kendall's tau on mention frequencies computed from the single ontology produced by the unvalidated matcher. The paper reports no confidence intervals, repeated stochastic runs, or robustness checks across ontology construction parameters, so the significance statements in Tables 9 and 10 are conditional on an unverified intermediate output. At minimum, the authors should report the variability of trend classifications across repeated runs and across the unspecified match threshold.","section":"Section 5.1, Tables 9 and 10"}],"minor_comments":[{"comment":"The word 'restructring' should be 'restructuring.'","section":"Table 9"},{"comment":"The text 'We chooseEVsandSemiconductorsas our sectors' contains a typo: 'EVs' is rendered as 'EVSs.'","section":"Section 2"},{"comment":"The entry '1.000.25' appears to be a formatting artifact; it should read '1.00 0.25' (and similarly in Table 7).","section":"Tables 6 and 7"},{"comment":"The 32 root and 361 second-level seed topics are introduced without provenance; a brief description of how they were chosen and whether the results are robust to this choice would help.","section":"Section 4.2"},{"comment":"The claim that insights 'appear significantly earlier' than financial media is reported without quantitative timestamps or a systematic comparison; this should be presented as anecdotal or supported with dates.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style paper with a plausible application demo. The main gap is that the evaluation does not go beyond internal consistency and post-hoc qualitative checks. I would require at least a stability/sensitivity analysis and a small expert evaluation to support the central claim. If the authors can add these, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a useful practical demonstration of an LLM-agent pipeline for building and maintaining a topic ontology over earnings calls, but the evaluation is mostly internal consistency and the one number anyone would quote (the 0.383 coherence) rests on a single stochastic run with an unspecified semantic-match threshold. The stress-test note is right.\n\nWhat is actually new: the combination of a topic retriever, an alias-based semantic matcher, and a tree-structured ontology maintained by an 'ontologist' agent, applied to quarterly earnings calls. The component techniques are all known—LLM topic extraction, ontology generation, trend detection—but the integrated system plus the alias mechanism for merging synonymous topics is a fair novelty claim for a workshop paper. The writing is clear and the examples in Tables 1, 9, and 10 are plausible and readable. The paper also does the right thing by testing against LDA and reporting a coherence check; the LDA comparison is weak but at least present.\n\nThe soft spots are in proportion to the claims. The central claim of 'dynamically capturing emerging topics' is not yet demonstrated because the entire ontology is built in one pass with a decision threshold that is never stated. Appendix A.3 asks the LLM for similarity scores but the paper never says what score counts as a match. So the 3,200 nodes and the alias statistics are artifacts of an unknown rule, and every downstream trend and emerging-topic table inherits that. A stability analysis over a few runs with different thresholds and temperatures is essential before the system can be called reliable. Second, the 'ontology coherence' evaluation measures the LLM's own grouping against the same LLM's embeddings; it shows the system is internally consistent, not that it matches an external truth. The seed ontology is also large and human-defined—393 pre-set topics—which is a bigger human input than the 'agentic discovery' framing suggests. Third, the trend validation is anecdotal: selecting examples that match public reports after the fact is not a test. The paper honestly concedes most of this in the Limitations section, which counts for something, but the missing threshold and missing runs are concrete, fixable omissions.\n\nWho this is for: people in financial NLP or applied LLM agents who want a prototype of dynamic topic tracking. It deserves a serious referee, especially as a workshop paper, but the decision should be conditional on adding a sensitivity analysis and specifying the match threshold.","headline":"A plausible agentic topic-ontology pipeline for earnings calls, but the evaluation rests on a single run with an unspecified matching threshold; the paper is honest about limits and deserves revision, not rejection.","tokens_in":11654,"tokens_out":2266,"would_cite":false,"duration_ms":33325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that an LLM-driven agentic pipeline—topic retriever plus ontologist—can extract financially relevant topics from earnings calls and organize them into a hierarchical ontology that stays current as new…","keywords":["Topics","Ontology","Insights","Retrieval","Financial documents","Earnings Calls","LLM agents","Trend analysis"],"falsifier":"Take a random sample of, say, 500 paragraphs from the 141 transcripts, and have two financial analysts independently assign canonical topic labels and parent categories to each excerpt. If the LLM-retrieved topics disagree with the analysts on more than a small fraction of paragraphs—or if the parent-child edges in the ontology are frequently judged wrong—then the ontology's coherence score and the downstream trend and competitor analyses lose their foundation.","tokens_in":10694,"feed_emoji":"📈","tokens_out":6909,"duration_ms":71263,"temperature":0.7,"pith_summary":"Earnings calls are dense sources of strategic information, but traditional topic models need a pre-vetted list of topics and often miss emerging themes. The paper proposes an LLM-agent framework that extracts topics directly from each paragraph, checks whether a topic already exists via semantic equivalence, and if not, inserts it into a tree-structured ontology under the most specific matching parent. This lets the topic hierarchy grow with the data instead of being fixed in advance. The authors show the constructed ontology is internally coherent, that mention-frequency trends align with known industry shifts, and that the framework surfaces emerging topics in EV and semiconductor sectors that analysts would want to track. The value, if sustained, is a scalable way to monitor strategic narratives across companies and quarters without manual topic curation.","feed_headline":"LLM agents map earnings calls into a live topic tree","feed_subtitle":"System surfaces rising and falling financial themes without a pre-vetted topic list.","key_machinery":"The central mechanism is the semantic-equivalence check performed by the Ontologist agent before any topic is added. The agent is prompted to determine whether a candidate topic matches an existing node or alias with bidirectional equivalence—same scope and specificity—rather than treating it as a parent or subset category. This single gate controls the entire ontology: it decides whether new topics are named consistently, whether aliases accrue, and whether parent-child relations stay at comparable granularity. Supporting it are the Topic Retriever's paragraph-level extraction prompt and a tree ontology where each node stores a UUID, a name, aliases, and creation/update timestamps. The coherence evaluation, which measures embedding similarity between parent and child nodes against random pairings, provides the paper's quantitative check on the whole construction.","core_discovery":"On its own terms, this paper establishes that a two-agent LLM pipeline can discover and maintain a hierarchical topic ontology from earnings-call transcripts. A Topic Retriever reads paragraph-level text and outputs financially relevant topic names with short excerpts; an Ontologist agent then decides, by semantic equivalence, whether a retrieved topic is already present—and if not, determines the narrowest parent node under which to insert it. Running this over 141 transcripts from 12 companies across EV and semiconductor sectors yields an ontology of 3,200 nodes with a median of 2.5 aliases per node and a four-level depth. The paper validates the ontology by showing that parent-child pairs have substantially higher embedding similarity than random pairings, and it demonstrates downstream utility by identifying statistically significant rising and falling topics (e.g., 'Supply Chain' declining, 'Generative AI' rising) and by comparing companies via Jaccard similarity over their top-100 topics.","pith_inferences":["The paper's own limitation statement admits there is no gold-standard annotation for topics; a direct implication is that the ontology's reliability should be measured against expert labels on a sample of paragraphs before production use in high-stakes financial decisions.","Because the entire analysis rests on the LLM's topic naming and matching, a cheaper or more deterministic alternative (e.g., a small fine-tuned classifier for existence checks) could achieve the same ontology-growth behaviour with less cost and variance; the paper does not test this.","The trend analysis uses Kendall's tau on raw mention counts; a natural extension would be to test whether the same significance holds when counts are normalized by call length or when model-blinded to the sector, to rule out confounds like call duration.","If the framework is correct, a strict out-of-time test—building the ontology on 2021–2023 calls and asking whether it surfaces topics that 2024 calls actually discuss—would directly validate its emerging-topic claims; the paper only shows post-hoc identified trends."],"forward_implications":["Financial analysts can track how often and how directionally topics are mentioned quarter by quarter, spotting trends like the decline of 'Supply Chain' discussions as soon as the call is released.","The hierarchical ontology with aliases lets retrieval recognize synonymous topic names ('M&A' and 'Mergers & Acquisitions'), so downstream queries on a theme aggregate mentions that would otherwise be scattered.","The same pipeline can be pointed at new sectors or document collections without retraining a supervised topic model, as long as the LLM prompt generalizes.","Comparing competitors by shared top topics (Jaccard similarity over leaf nodes) gives a qualitative complement to financial ratios, exposing strategic focus as well as performance.","If the ontology stays coherent as it grows, emerging-topic lists such as 'on-premises AI,' 'AI copilots,' and 'low-cost vehicles' can feed early signal for equity research."],"supporting_citations":[{"why":"Establishes why earnings-call tone matters for financial performance, motivating the dataset selection.","marker":"[1]"},{"why":"Earlier LLM-based topic and sentiment modeling for finance, which the paper contrasts with and builds upon.","marker":"[7]"},{"why":"Latent Dirichlet Allocation, the standard unsupervised baseline the paper benchmarks against qualitatively.","marker":"[8]"},{"why":"Supports the use of LLMs for ontology generation, grounding the ontologist agent's design.","marker":"[9]"},{"why":"Provides the open-source LDA workflow adapted to earnings-call transcripts for the baseline comparison.","marker":"[10]"},{"why":"Supplies the sentence-embedding model used to measure parent-child semantic coherence in the ontology.","marker":"[11]"},{"why":"Gives the Kendall's tau procedure used to detect statistically significant topic trends over time.","marker":"[12]"},{"why":"Cited in the limitations as an example of neural topic modeling for emerging trends that also lacks gold-standard benchmarks.","marker":"[13]"}],"fun_headline_variants":["LLM agents build live topic maps from earnings calls","Earnings calls auto-mapped into dynamic topic hierarchies","AI agents surface rising and falling financial themes","Topic mining on earnings calls goes agentic and live"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's topic extraction and semantic-matching decisions are reliable enough that the ontology stays coherent and the resulting mention counts mean what they appear to mean; the paper acknowledges it has no gold-standard annotations to verify this.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents build live topic maps from earnings calls","Earnings calls auto-mapped into dynamic topic hierarchies","AI agents surface rising and falling financial themes","Topic mining on earnings calls goes agentic and live"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1264,"prompt_tokens":828,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":375}},"tokens_in":444,"tokens_out":436,"duration_ms":4805,"temperature":1.0,"reasoning_tokens":375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:29:04.684010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 500 paragraphs from the 141 transcripts, and have two financial analysts independently assign canonical topic labels and parent categories to each excerpt. If the LLM-retrieved topics disagree with the analysts on more than a small fraction of paragraphs—or if the parent-child edges in the ontology are frequently judged wrong—then the ontology's coherence score and the downstream trend and competitor analyses lose their foundation.","supporting_citations":[{"cited_title":"Managers set the tone: Equity incentives and the tone of earnings press releases.Journal of Banking & Finance, 72:S132–S147, 2016","cited_arxiv_id":null,"evidence_quote":"Establishes why earnings-call tone matters for financial performance, motivating the dataset selection."},{"cited_title":"Distilled ChatGPT Topic & Sentiment Modeling with Applications in Finance","cited_arxiv_id":"2403.02185","evidence_quote":"Earlier LLM-based topic and sentiment modeling for finance, which the paper contrasts with and builds upon."},{"cited_title":"Packt Publishing Ltd, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the open-source LDA workflow adapted to earnings-call transcripts for the baseline comparison."},{"cited_title":"Practical guide to using kendall’s 𝜏 in the context of forecasting critical transitions.Royal Society Open Science, 9(7):211346, 2022","cited_arxiv_id":null,"evidence_quote":"Gives the Kendall's tau procedure used to detect statistically significant topic trends over time."}],"review_version":1}