REVIEW 4 cited by
Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The hallucination issue is recognized as a fundamental deficiency of large language models (LLMs), especially when applied to fields such as finance, education, and law. Despite the growing concerns, there has been a lack of empirical investigation. In this paper, we provide an empirical examination of LLMs' hallucination behaviors in financial tasks. First, we empirically investigate LLM model's ability of explaining financial concepts and terminologies. Second, we assess LLM models' capacity of querying historical stock prices. Third, to alleviate the hallucination issue, we evaluate the efficacy of four practical methods, including few-shot learning, Decoding by Contrasting Layers (DoLa), the Retrieval Augmentation Generation (RAG) method and the prompt-based tool learning method for a function to generate a query command. Finally, our major finding is that off-the-shelf LLMs experience serious hallucination behaviors in financial tasks. Therefore, there is an urgent need to call for research efforts in mitigating LLMs' hallucination.
Forward citations
Cited by 4 Pith papers
-
Hallucination Detection in Large Language Models Using Diversion Decoding
Forcing an LLM away from its greedy answer yields resistance features that train a classifier detecting hallucinations more accurately and cheaply than semantic entropy.
-
Proof-Carrying Numbers (PCN): A Protocol for Trustworthy Numeric Answers from LLMs via Claim Verification
Proof-Carrying Numbers binds each displayed LLM number to a structured claim and only marks it verified after a policy-based mechanical check, leaving all other numbers unverified.
-
ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking
FAITH masks numbers in real 10-K reports to test when financial LLMs hallucinate, and finds even top models err on 10-20% of multi-step calculations.
-
LLM Harms: A Taxonomy and Discussion
This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.
Discussion (0). Sign in to comment.