Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance

Charlie Goldenberg; Derleen Saini; Dominick Kubica; Dylan T. Gordon; Nanami Emura

REVIEW 4 major objections 6 minor 1 cited by

Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims LLMs outperform traditional sentiment tools on financial text, and that segment-level earnings-call tone reveals stock-moving patterns that overall tone misses.

desk verdict A useful practitioner benchmark with one genuinely interesting finding (Copilot 365 silently falls back to TextBlob), but the stats are too thin to support "clearly outperformed" and the earnings-call segmentation is unvalidated. read the letter →

arxiv 2505.16090 v1 pith:RWPGZGZM submitted 2025-05-22 cs.AI cs.CL

Dominick Kubica , Dylan T. Gordon , Nanami Emura , Derleen Saini , Charlie Goldenberg This is my paper · ORCID

classification cs.AIcs.CL

keywords financialsentimentanalysislargelanguagemodelsearningscalltranscriptsbenchmarkingbusiness-linesegmentationSHAPvaluespromptengineeringstockpriceprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can interpret the hedged, forward-looking language of earnings calls well enough to be useful in finance. It answers by benchmarking nine tools on a standardized financial phrase dataset, finding that LLM-based systems, led by the Copilot App at 82.0% accuracy, outperform lexicon-based libraries, which lose signal when text is aggressively cleaned. It then applies ChatGPT-4o to quarterly earnings call transcripts segmented by business line and argues that sentiment patterns inside specific segments align with stock moves in ways overall transcript tone does not. If correct, practitioners should prefer LLM-based sentiment tools over traditional libraries and should segment earnings calls by business line to find market-relevant signals.

What carries the argument

The analysis is carried by three linked components: a standardized financial sentiment benchmark used as the accuracy yardstick, a segmentation step that splits each quarter's earnings call transcript by business line before sentiment scoring, and SHAP values that map each segment's sentiment to its contribution to a stock-price-change prediction. The benchmark establishes the model rankings, the segmentation is what exposes per-segment sentiment signals that aggregate tones obscure, and SHAP provides the quantitative link between segment tone and market response. The paper also relies on identical prompts across LLMs and on prompt engineering to keep the comparison fair.

What would settle it

Run the nine benchmark systems on a fresh, expert-labeled corpus of earnings-call sentences: if standard lexicon tools match or beat the LLMs' accuracy, the performance gap collapses. Separately, redo the segment-level SHAP analysis using human-verified business segments: if the Search and News Advertising inversion and the Devices spike no longer align with stock moves, the business-line insight is an artifact of the segmentation.

Watch

Extended reading notes

Core claim

Across all benchmark comparisons, the paper's central finding is that LLM-based tools read financial tone more accurately than traditional lexicon-based libraries. On the Financial Phrase Bank, the Copilot App (both online and local) reached 82.0% accuracy, ChatGPT-4o reached 77.6%, prompt-engineered ChatGPT 75.6%, and Gemini 2.0 Flash 68.0%, while NLTK, TextBlob, and a Copilot 365 configuration that silently fell back to TextBlob scored lower. The paper attributes the gap to text cleaning: traditional models strip away filler and hedged words that carry tone, while LLMs use that context. A second finding comes from the earnings-call case study: after segmenting transcripts by business line and scoring each segment with ChatGPT-4o, sentiment in specific segments such as Search and News Advertising and Devices aligned with post-call stock moves, while overall transcript sentiment did not. The paper interprets the SHAP beeswarm plot as evidence that positive tone in some segments can coincide with falling prices, so tone should not be read as a simple stock predictor.

Load-bearing premise

The business-line findings depend on the automatic splitting of each earnings call transcript into business segments being correct, and the paper does not validate that splitting against human judgment.

Editorial extensions

If this is right

Practitioners should prefer LLM-based sentiment tools over lexicon libraries for hedged or forward-looking financial text, but should expect accuracy below 85% and plan for verification.
Analysts should segment earnings call transcripts by business line before scoring sentiment, because aggregate tone can miss or even invert the market signal.
A positive tone in a specific segment is not reliably a positive stock signal; context and expectations can make upbeat language read as over-optimism, as with Search and News Advertising.
Deployment choices matter: Copilot 365's fallback to a simpler sentiment engine and CSV-handling issues can silently lower quality, so users need to know which engine is actually running.
Human domain expertise remains necessary for framing questions, choosing analyses, and checking outputs; LLMs support but do not replace the analyst.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If segment-level sentiment is genuinely predictive, the same segmented design could be applied to earnings calls of other large firms, and firm-specific tell segments might emerge that general sentiment models would miss.
The observed inversion (positive tone, falling stock) invites a testable explanation: upbeat sentiment in a segment may signal that good news is already priced in, and conditioning segment sentiment on prior analyst expectations could confirm this.
Because the business-line segmentation was not independently validated, an audit that re-segments a few quarters by human labelers and re-runs the SHAP analysis would show whether the stock correlations hold up.
Combining LLM sentiment with finance-specific tools such as FinBERT could push past the 85% ceiling, since the paper's own data show different error patterns across model families.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I read this as an honest industry practicum report, not a research paper, and it is most useful when read that way. What is genuinely new here is narrow but real: the discovery that Copilot 365 silently routes sentiment analysis to TextBlob, the accuracy numbers for Copilot variants on a standardized financial dataset, and the business-line segmentation of Microsoft earnings calls. The TextBlob fallback is the kind of concrete, reproducible observation that practitioners will actually care about, and I give them credit for flagging it instead of burying it.

The paper does some things well. It is readable, the benchmark design is simple, and the final section properly hedges the stock-movement claims and says LLMs are not replacements for experts. The exploratory framing of the SHAP analysis is appropriate on the surface.

Now the soft spots, in proportion. The central claim in Section 2 that "LLMs clearly outperformed traditional tools" is not statistically supported. The headline numbers come from the first 250 rows of one dataset, with no confidence intervals, no significance tests, and no disclosure of the exact prompts or model settings. The gap between the Copilot App at 82.0% and ChatGPT-4o at 77.6% is within plausible sampling noise, and even the lead over FinBERT is not obviously outside it. I am not saying the direction is wrong; I am saying the evidence is thinner than the word "clearly" allows. The case study has a separate fragile step: Copilot's segmentation of transcripts by business line is unvalidated, and the SHAP beeswarm is computed from sentiment values that inherit any segmentation error. The "Search and News Advertising inversion" and the "Devices spike" are interesting hypotheses, but they are post-hoc stories on a handful of quarters, not findings. The authors' own conclusion is appropriately tentative, which helps, but the abstract and the early sections push the claims further than the methods support.

The citation pattern is thin for a benchmark paper: two references, one being a Kaggle dataset, with no comparison to prior work on LLM financial sentiment or to earlier findings on business-line tone. That is a minor issue if the paper is positioned as a practical report, but it would matter in a peer-reviewed venue.

Who is this for? Practitioners choosing between Copilot variants and open libraries for financial sentiment work, and people studying how LLM-based tools actually behave in production settings. It does not deserve a hard desk reject—the TextBlob fallback alone justifies a referee’s time—but it needs a serious revision to add statistical rigor, release the prompts and data subset, and validate the segmentation before the business-line conclusions can be trusted. I would not cite it in my own work as evidence for a claim beyond the TextBlob observation, and I would not bring it to a reading group for its methods, but I would engage with it as a useful field report.

Referee Report

4 major / 6 minor

Summary. The paper reports a benchmark of nine sentiment-analysis tools (Microsoft Copilot variants, ChatGPT-4o, Gemini, Azure, FinBERT, NLTK, TextBlob) on the first 250 rows of the Financial Phrase Bank, and a case study applying ChatGPT-4o to Microsoft earnings-call transcripts segmented by business line using Microsoft Copilot. The authors claim that LLMs, especially the Copilot App, clearly outperformed traditional tools in detecting nuanced financial sentiment, and that segment-level sentiment, visualized with a SHAP beeswarm plot, reveals business-line-specific relationships with stock price movements. The manuscript is written as a project report rather than a full research paper, and its central claims rest on point-estimate accuracy figures and an exploratory SHAP analysis without reported statistical support or validation of the transcript-segmentation step.

Significance. If the benchmark claims were statistically supported, the paper would offer practically useful guidance for practitioners selecting sentiment tools for financial text: it directly compares several widely used cloud LLM interfaces and Python libraries on a standard dataset, and it attempts to extend sentiment analysis from whole-transcript tone to business-line-level tone. The real-world case study, despite being exploratory, illustrates a plausible workflow that could be valuable if validated. The paper also gives credit for documenting concrete failure modes of LLM deployment (CSV handling, fallback to TextBlob, hallucinations) and for explicit hedging in the conclusion that stock movement prediction remains out of reach. However, the current evidence is insufficient to establish the headline claims, and the lack of statistical inference and segmentation validation limits the paper's contribution to an industry anecdote rather than a reliable research result.

major comments (4)

[Section 2, Figure 1] The claim that LLMs 'significantly outperformed' traditional tools is not supported by the reported statistics. The benchmark reports only point accuracies on 250 sentences, with no confidence intervals, significance tests, or repeated trials. For n=250, the standard error of an 82.0% accuracy is about 2.4 percentage points, so the reported 4.4-point lead over ChatGPT-4o is within roughly two standard errors, and the differences against other tools are similarly plausible under sampling noise. The authors should provide per-model standard errors or exact binomial confidence intervals, a formal significance test that accounts for multiple comparisons, and a description of how the 250-row subset was selected.
[Section 3, transcript segmentation] The business-line analysis depends on Microsoft Copilot's segmentation of earnings-call transcripts into segments (Devices, Dynamics, Gaming, etc.), but the manuscript reports no validation of this segmentation. If sentences are misassigned to business lines, the segment-level sentiment scores and the subsequent SHAP-based interpretation would be unreliable. The authors should validate the segmentation on a random sample against human annotation, or at least report inter-coder or model-consistency metrics, and describe the exact instructions given to Copilot for this task.
[Section 3, Figure 4 (SHAP analysis)] The SHAP beeswarm plot is not reproducible and its interpretation is not statistically grounded. The manuscript does not specify the predictive model used to map sentiment values to stock price changes, the number of data points (segments by quarter), the feature set, or any train/test or stability analysis. With a small number of quarters and multiple segments, SHAP values are known to be unstable, and the claim that 'positive tone in Search and News Advertising is associated with stock decline' is a post-hoc reading of a few anecdotal quarters rather than a tested hypothesis. The authors should report the model formulation, sample size, and a permutation or bootstrap-based stability check, and explicitly label these findings as exploratory.
[Section 4, bullet list] The summary bullet 'LLMs clearly outperformed traditional tools in financial sentiment analysis' overstates the evidence. The benchmark is a single, unduplicated run on 250 items, and the paper's own Figure 1 shows a range of LLM accuracies that overlaps once realistic uncertainty is considered. Without significance tests or repeated trials, the correct claim is that the LLM tools achieved higher point estimates in this particular run. This language should be softened to match the statistical support.

minor comments (6)

[Section 2] The exact prompts, model settings, temperature parameters, and versions of each LLM are not disclosed, which limits reproducibility; provide a supplementary appendix with the full prompt and configuration used for each model.
[Section 2, Figure 2] The paper refers to a 'Condensed Sentiment Accuracy Comparison' but does not explain which models are included or what 'condensed' means; please clarify the subset of models and the metric depicted.
[Section 2, p. 3] The sentence 'FinBERT was particularly effective for finance-specific cases' is vague and potentially in tension with the earlier statement that LLMs outperformed traditional tools; please specify the quantitative basis for this claim.
[Section 2 and 4] The term 'filler words' is used to describe what are apparently hedged or nuanced expressions; consider using more precise terminology such as 'hedging phrases' or 'qualifying language' to avoid confusion.
[References] Reference [2] points to a Kaggle dataset rather than the original Financial Phrase Bank paper; please cite the original Aalto University publication and report the exact version of the dataset used.
[Section 3, Figure 3 and 4] The figures should include axis labels, units, and a caption describing the number of quarters, the source of the transcripts, and the exact stock-price return window; currently the reader must infer these details from the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark scores rest on external Financial Phrase Bank labels and stock returns, and no load-bearing self-citation or fitted-input prediction is present.

full rationale

The paper makes two main empirical claims: LLMs outperform traditional tools on financial sentiment, and business-line sentiment correlates with stock moves. The first claim is evaluated against the external Financial Phrase Bank labels ('Accuracy testing was conducted using the Financial Phrase Bank dataset... pre-labeled dataset'), so the target labels are not derived from the models being tested. The second claim uses ChatGPT sentiment scores from earnings transcripts and compares them with actual stock price changes after earnings calls ('The arrows below represent the stock price increase or decrease the day after the earnings call'), an independent external outcome. The SHAP beeswarm explains a stock-movement prediction model, not the sentiment scores themselves, so no quantity is being predicted from itself. The unvalidated Copilot segmentation and the absence of statistical significance tests are correctness and robustness concerns, not circularity. No fitted parameter is renamed as a prediction, no self-citation is load-bearing, and no derivation reduces to its input by construction. The paper is exploratory and methodologically limited, but its derivation chain is not circular.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the paper uses existing LLMs, a standard sentiment dataset, and a fitted SHAP model. The only fitted quantity is the unreported stock-change model, and the main assumptions concern data labeling, segmentation, and the market-reaction proxy.

free parameters (1)

SHAP stock-change model parameters = not reported
A model mapping business-line sentiment to next-day stock change is fit to the earnings-call data; coefficients are not reported, so the SHAP attributions in Figure 4 cannot be audited.

assumptions (4)

domain assumption Financial Phrase Bank labels are treated as ground truth for sentiment.
Benchmark accuracy is computed against these labels; the paper does not report inter-annotator agreement or label quality checks.
domain assumption Copilot's business-line segmentation of earnings call transcripts is accurate.
Section 3 uses this segmentation as the basis for all business-line sentiment scores, but no validation of segmentation quality is provided.
domain assumption The 'First 250 Rows' of the dataset are representative of the full dataset.
The subset is described only by its position in the file; no random sampling or representativeness check is reported.
domain assumption Next-day stock price change is a valid proxy for market reaction to the earnings call.
The arrows in Figure 3 and the SHAP target use this proxy without considering other market-moving events.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance." pith.science (2026). https://pith.science/paper/RWPGZGZM

@misc{pith2026250516090,
  author       = {Pith},
  title        = {Pith review of: Can AI Read Between The Lines? Benchmarking LLMs On Financial Nuance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWPGZGZM}},
  note         = {Machine review of arXiv:2505.16090}
}

read the original abstract

As of 2025, Generative Artificial Intelligence (GenAI) has become a central tool for productivity across industries. Beyond text generation, GenAI now plays a critical role in coding, data analysis, and research workflows. As large language models (LLMs) continue to evolve, it is essential to assess the reliability and accuracy of their outputs, especially in specialized, high-stakes domains like finance. Most modern LLMs transform text into numerical vectors, which are used in operations such as cosine similarity searches to generate responses. However, this abstraction process can lead to misinterpretation of emotional tone, particularly in nuanced financial contexts. While LLMs generally excel at identifying sentiment in everyday language, these models often struggle with the nuanced, strategically ambiguous language found in earnings call transcripts. Financial disclosures frequently embed sentiment in hedged statements, forward-looking language, and industry-specific jargon, making it difficult even for human analysts to interpret consistently, let alone AI models. This paper presents findings from the Santa Clara Microsoft Practicum Project, led by Professor Charlie Goldenberg, which benchmarks the performance of Microsoft's Copilot, OpenAI's ChatGPT, Google's Gemini, and traditional machine learning models for sentiment analysis of financial text. Using Microsoft earnings call transcripts, the analysis assesses how well LLM-derived sentiment correlates with market sentiment and stock movements and evaluates the accuracy of model outputs. Prompt engineering techniques are also examined to improve sentiment analysis results. Visualizations of sentiment consistency are developed to evaluate alignment between tone and stock performance, with sentiment trends analyzed across Microsoft's lines of business to determine which segments exert the greatest influence.

Figures

Figures reproduced from arXiv: 2505.16090 by the authors.

**Figure 2.** Condensed Sentiment Accuracy Comparison (First 250 Rows) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Positive Sentiment by Business Line - ChatGPT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: SHAP Beeswarm: Effect of Net Sentiment on Stock Prediction [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Quantitative Analysis of Media Bias and Stock Price Dynamics: The 2020 Shock
cs.CE 2026-08 reject novelty 6.0 of 10

Media stance and stock returns show no market-wide level shift or Granger-causal link around 2020; significant links appear only for individual firms after their own structural breaks.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · cited by 1 Pith paper

[1]

Investor relations.https://www.microsoft.com/en-us/investor/default, 2025

Microsoft Corporation. Investor relations.https://www.microsoft.com/en-us/investor/default, 2025. Accessed: 2025-05-08

work page 2025
[2]

S. Bhatti. Financial sentiment analysis. https://www.kaggle.com/datasets/sbhatti/financial-sentiment-analysis, 2021. Accessed: 2025-05-08. 6

work page 2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Investor relations.https://www.microsoft.com/en-us/investor/default, 2025

Microsoft Corporation. Investor relations.https://www.microsoft.com/en-us/investor/default, 2025. Accessed: 2025-05-08

work page 2025

[2] [2]

S. Bhatti. Financial sentiment analysis. https://www.kaggle.com/datasets/sbhatti/financial-sentiment-analysis, 2021. Accessed: 2025-05-08. 6

work page 2021