Pith. sign in

REVIEW 4 major objections 4 minor 30 references

Sentiment Analysis of Economic Text: A Lexicon-Based Approach

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a sentiment lexicon built from economic text and scored by human annotators yields a pessimism measure that dominates three established dictionaries in explaining uncertainty, consumer sentiment, and recession…

desk verdict A genuinely useful and reproducible lexicon resource, but the superiority claim over existing dictionaries is not yet established due to in-sample evaluation and a misread AUC test. read the letter →

arxiv 2411.13958 v1 pith:BYKQJQRI submitted 2024-11-21 cs.CE cs.CLcs.CY

Luca Barbaglia , Sergio Consoli , Sebastiano Manzan , Luca Tiozzo Pezzoli , Elisa Tosetti This is my paper · ORCID
classification cs.CEcs.CLcs.CY
keywords economiclexiconsentimentanalysistextualpessimismhumanannotationdependencyparsingrecessionforecastingdictionarycomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a sentiment dictionary, the Economic Lexicon (EL), from a large corpus of economic news and central bank documents, and then argues that a simple text-based measure of economic pessimism built from it outperforms measures built from three established lexicons. The EL selects words by parsing sentences that discuss economic concepts and has human annotators score each word on a scale from -1 to 1. The paper shows that the EL-based pessimism index correlates with financial and policy uncertainty and consumer sentiment, and helps predict recessions; in joint regressions it makes the other lexicons' measures statistically irrelevant. The authors attribute this advantage to wider coverage of economics-specific terms and a more accurate assignment of word sentiment than the alternatives. A sympathetic reader would care because the result offers a reusable, human-annotated tool for measuring tone in economic text.

What carries the argument

The central object is the EL lexicon itself, together with the Economic Pessimism (EP) measure that turns it into a time series. Construction follows four steps: sentences are selected only if they contain one of 355 economic entities drawn from an economic topical taxonomy and central bank documents; dependency parsing extracts adverb, verb, and adjective modifiers of those entities, keeping words that occur at least 65 times; ten human annotators assign each term a sentiment score in [-1,1], and the median score becomes the word's tone; and words flagged as ambiguous are removed, leaving 6,670 terms. The EP measure is the negative of the frequency-weighted sum of term scores, so positive values indicate that negative words outnumber positive words in a month. This pipeline carries the argument because it is designed to include words that co-occur with economic concepts and to avoid finance-only or general-purpose words, and the fine-grained human scores allow both signed categorization and strength-sensitive sentiment measurement.

What would settle it

Compute the EL-based pessimism index and the three rival indices on an out-of-sample corpus never used in lexicon construction, for example central bank speeches or non-US news from after 2020, and test whether EL still dominates in explaining VIX, EPU, consumer sentiment, and recession forecasts. If the EL advantage shrinks or reverses under this test, the claimed general superiority would be refuted.

Watch

Extended reading notes

Core claim

The authors' central claim is that the Economic Lexicon, constructed from economic text rather than general-purpose or finance-only text, produces a sentiment measure that is more accurate for economics than the Loughran-McDonald financial dictionary, the Renault model-based lexicon, and the Shapiro-Sudhof-Wilson news lexicon. After extracting noun phrases tied to 355 economic concepts from over 13 million newspaper articles and central bank documents, dependency parsing yields modifier words; frequency filtering and human annotation leave 6,670 terms with median sentiment scores on [-1,1] and ambiguous terms removed. An Economic Pessimism index formed as the frequency-weighted sum of these scores is significantly related to VIX, economic policy uncertainty, and the Michigan Consumer Sentiment Index, and adds predictive power for NBER recessions. Once the EL-based index is included in regressions alongside the other lexicons' indices, the alternatives lose significance. The paper attributes this result to two factors: EL includes more sentiment-bearing terms that actually appear in economic discussion, and its human-annotated scores correct the sign or strength of words that the other lexicons misclassify.

Load-bearing premise

The load-bearing premise is that the corpus used to select EL's words is representative of economic text generally; the comparisons are run on the same six US newspapers (1980-2020) from which the vocabulary was extracted, so if those newspapers are not representative, EL's apparent edge could be an artifact of in-sample word selection.

Editorial extensions

If this is right

  • Using the EL, a researcher can compute economic sentiment from news or central bank text without training a model-based sentiment classifier, because the lexicon ships with human median scores on a continuous scale.
  • In the paper's regressions, the EL-based pessimism measure absorbs the explanatory power of the LMD, REN, and SSW measures for VIX, EPU, consumer sentiment, and recessions at a three-month horizon, so those rival indexes add no significant information once EL is included.
  • The finding that positive and negative economic words appear with similar frequency suggests sentiment indexes built only from negative words may miss a comparable share of sentiment variation, and the EL supplies positive terms for that purpose.
  • Re-scoring disagreeing terms with EL values or adding EL-only terms improves the recession-forecasting performance of the other lexicons, implying that both word coverage and sentiment assignment contribute to EL's edge.
  • Because EL scores are continuous, the same lexicon can be used for simple signed counts and for strength-weighted sentiment measures such as the EP index.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same-corpus evaluation in the paper is the main threat to generalization; an out-of-sample corpus test would settle whether EL's dominance is a property of the lexicon or of the newspapers used to build it.
  • My inference: the dependency-parsing-plus-human-annotation pipeline could be transferred to other specialized domains, such as climate finance or health policy, where general sentiment dictionaries are known to misfire.
  • My inference: the fine-grained human scores might also serve as a benchmark to audit model-based sentiment classifiers, since the paper shows machine and human scores agree in sign on most but not all shared words.
  • My inference: the paper's evidence that negative-word sentiment is more volatile and procyclical than positive-word sentiment implies that future sentiment measures could weight the two components separately rather than forcing them into a single pessimism index.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a new domain-specific Economic Lexicon (EL) for sentiment analysis in economics, built from a large corpus of news articles and central bank documents. The lexicon is constructed by extracting noun phrases around a list of economic concepts, selecting frequent modifier words, and assigning sentiment scores via human annotators, with ambiguous terms removed. The EL is then compared with three existing lexicons (LMD, REN, SSW) in several applications: explaining movements in the VIX and economic policy uncertainty, correlating with consumer sentiment, and forecasting NBER recessions using an Economic Pessimism (EP) measure. The authors claim that EL is superior because it covers more economically relevant terms and categorizes sentiment more accurately, and that its EP measure is significant and makes alternative EP measures irrelevant when included jointly.

Significance. If the claims were established, the paper would provide a valuable, freely available economic sentiment lexicon with fine-grained human-annotated scores, and a transparent construction pipeline. The authors also make available a replication package, which is a commendable feature. However, the central superiority claim is not established by the current evidence: the evaluation is performed largely on the same corpus used to select the lexicon's vocabulary, the word-level 'accuracy' claim is benchmarked only against the annotators' own scores, and the paper's own footnote 19 contradicts the introduction's dominance claim. The resource itself is a useful contribution, but the empirical evidence for its general superiority needs substantial additional work before the claim can be accepted.

major comments (4)
  1. [Section 4.2 and Section 6] The evaluation is in-sample by construction. The EL vocabulary is selected by applying a frequency threshold of 65 occurrences to the corpus of six US newspapers over 1980-2020 (Section 4.2), and the EP measure used for all comparisons in Section 6 is computed on exactly the same six newspapers over the same period. This creates a selection loop: words are chosen because they are frequent in this corpus and then the EP measure is shown to be informative in this same corpus. The superior performance in Tables 2, 3, and 5 may therefore reflect frequency-based overfitting rather than a general property of the lexicon. The authors should provide an out-of-sample evaluation on text not used in the construction (e.g., UK newspapers, central bank documents, or a temporal holdout such as constructing the lexicon on pre-2000 data and evaluating on post-2000 data). The oversampling of negative terms during recessions noted in Section 4.2 makes this concern more acute, as the in-sample evaluation could inherit a pro-cyclical negative-word bias.
  2. [Section 7, footnote 19] There is an internal contradiction between the claim in the Introduction that 'once included, makes the other EP measures irrelevant' and the statement in footnote 19 that when EP_EL is added as an additional regressor to model (5), 'only EP_LMD,t retains its significance.' This implies that LMD contains predictive content not captured by EL, which directly undermines the irrelevance claim. The authors must either reconcile these findings or substantially revise the claim. As written, the paper simultaneously asserts dominance and provides evidence against it.
  3. [Section 5 and Section 4.3] The claim that EL provides 'more accurate categorization of the word sentiment' is not supported by an external benchmark. The word-level 'accuracy' is assessed against the same human annotator scores that define the lexicon (Section 4.3), and Section 5 explicitly notes that no human-annotated sentence-level scores are available to benchmark the lexicons. The comparison to other lexicons is thus an agreement study, not a validation against a gold standard. A human-annotated benchmark such as the 800 articles used in Shapiro et al. (2022) should be used to ground the accuracy claim; Section 7 acknowledges this as future work, but it is load-bearing for the paper's central claim.
  4. [Section 6.3, Figure 9] The discussion of the AUC results overstates what the test shows. The p-values in Figure 9 are for the null hypothesis that the AUC of the EL-based model is larger than or equal to the alternative's AUC; failing to reject this null does not establish that EL is superior. The text states that 'at most horizons' EL provides 'better performance, at least in terms of larger AUC,' but the only horizon at which the null is rejected is h = 2 for LMD. This should be described as a non-inferiority result, not a superiority result, and the conclusions in Sections 1 and 6.4 should be adjusted accordingly.
minor comments (4)
  1. [Figure 8 caption] The caption refers to 'RLM' where the intended dictionary is LMD (Loughran-McDonald); this spelling should be corrected.
  2. [Section 1, last paragraph] The phrase 'makes the other EP measures irrelevant' is too strong even apart from footnote 19; a more precise statement would say that EP_EL is significant when included jointly with each alternative, as in Tables 2, 3, and 5, while noting that the joint inclusion of more than one alternative is not reported.
  3. [Section 6, Equation (1)] The notation Si,l is defined as the sentiment score of word i in lexicon l, but for the fine-grained lexicons the equation uses Si,l directly even though the text says the scores were converted to ±1 for comparability; clarifying this, as is done in footnote 13, would improve readability.
  4. [Data availability] The data availability statement references the replication package for the related paper by Barbaglia et al. (2024) at openICPSR; it would be helpful to state explicitly whether this package includes the EL lexicon and the code for the current paper's tables and figures.

Circularity Check

1 steps flagged · score 5.0 of 10

EL's superiority claim rests on an in-sample evaluation loop: the lexicon's word list is selected from the same corpus on which its EP measure is tested.

  1. fitted input called prediction [Section 4.2 (word selection) and Section 6 (EP computation)]
    "Since many of these terms appear very infrequently, we decided to narrow down the list to only words that occur at least 65 times. ... We calculate the EPl,t measure on all economic-relevant phrases extracted from news published in the six main US newspapers mentioned earlier from January 1st, 1980 to December 31st, 2020."

    The EL vocabulary is selected by a frequency threshold applied to the same six-newspaper 1980-2020 corpus on which EP_EL is later computed and compared with LMD, REN, and SSW. Consequently, the claimed 'wider coverage' of EL is true largely by construction: any sentiment word that is frequent in this corpus is a candidate for inclusion, whereas lexicons constructed from other corpora will miss corpus-specific vocabulary. The evaluation is in-sample and does not establish general coverage. The paper itself notes in Section 4.2 that economic sentences are oversampled during recessions, which biases the selected negative vocabulary toward recession language and makes EP_EL's recession-forecasting edge partly an artifact of the construction corpus.

full rationale

The main circularity is an in-sample evaluation loop: the EL's word list is selected by a 65-occurrence frequency cutoff on the same six-US-newspaper 1980-2020 corpus on which EP_EL is later computed and benchmarked against lexicons built from other text sources. The coverage advantage is therefore partly guaranteed by construction, and the paper's own admission that economic sentences are oversampled in recessions makes the recession-prediction edge partly an artifact of the construction corpus. The human-annotated sentiment scores are not fitted to recession outcomes, which gives the lexicon some independent content, so the circularity is moderate rather than total. A separate consistency problem exists but is not circularity: footnote 19 states that when EP_EL is added to model (5), only EP_LMD retains significance, directly contradicting the Introduction's claim that EP_EL makes other EP measures irrelevant; that is a robustness concern, not a derivation loop.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several hand-set thresholds and domain assumptions rather than on fitted parameters. The free parameters are construction choices, not tuned to the outcome variables, but they materially shape the lexicon. No new physical or theoretical entities are introduced.

free parameters (4)
  • minimum word frequency threshold = 65 occurrences
    Words occurring fewer than 65 times in the corpus were excluded (Section 4.2). The threshold was chosen mostly by annotation budget, not by a data-driven criterion.
  • sentiment relevance agreement threshold = at least 2 of 4 author annotators
    Words considered sentiment-bearing by fewer than two of four authors were dropped (Section 4.2), a subjective rule.
  • ambiguity removal threshold = 713 terms removed
    Terms flagged as ambiguous by at least one of five authors were removed (Section 4.4), a subjective and discretionary filter.
  • number of economic entities = 355 entities
    The seed list of economic concepts was hand-compiled from the WBG taxonomy and central bank documents (Section 4.1).
assumptions (4)
  • domain assumption Dependency parsing around 355 economic entities identifies the words most relevant to economic sentiment.
    Section 4.2: modifier words extracted from noun phrases whose head is an economic concept are assumed to capture sentiment-bearing vocabulary.
  • domain assumption The median of 10 US annotator scores yields an accurate sentiment score for each word.
    Section 4.3: no external gold standard is used; annotator consensus is treated as ground truth.
  • domain assumption The 13 million Factiva articles plus central bank documents are representative of economic text.
    Section 3: corpus composition determines which words are frequent enough to enter the lexicon.
  • domain assumption Economic Pessimism, defined as the negative of the average signed frequency, is a valid sentiment measure.
    Section 6, Equation (1): the paper uses this simple linear measure for all comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sentiment Analysis of Economic Text: A Lexicon-Based Approach." pith.science (2026). https://pith.science/paper/BYKQJQRI

@misc{pith2026241113958,
  author       = {Pith},
  title        = {Pith review of: Sentiment Analysis of Economic Text: A Lexicon-Based Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYKQJQRI}},
  note         = {Machine review of arXiv:2411.13958}
}
read the original abstract

We propose an Economic Lexicon (EL) specifically designed for textual applications in economics. We construct the dictionary with two important characteristics: 1) to have a wide coverage of terms used in documents discussing economic concepts, and 2) to provide a human-annotated sentiment score in the range [-1,1]. We illustrate the use of the EL in the context of a simple sentiment measure and consider several applications in economics. The comparison to other lexicons shows that the EL is superior due to its wider coverage of domain relevant terms and its more accurate categorization of the word sentiment.

Figures

Figures reproduced from arXiv: 2411.13958 by the authors.

Figure 1
Figure 1. Time series of the number of noun phrases extracted aggregated at the monthly [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Scatter plots of the sentiment scores of EL against LMD, REN and SSW dictio [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Scatter plots of the sentiment scores of EL against LMD, REN and SSW dictio [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Scatter plots of the sentiment scores of EL against REN and SSW dictionaries at [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Time series of the number of positive and negative words for the EL, LMD, REN, [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Time series of Economic Pessimism by lexicon (i.e., EPEL, EPLMD, EPSW and EPREN ) at the monthly frequency. The series are standardized to have mean zero and variance one, and smoothed with a three-month rolling moving average to appreciate the differences among indica…
Figure 7
Figure 7. Figure 7: EPU and VIX (blue solid line in the top and bottom panels, respectively) plotted [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: Michigan Consumer Sentiment Index (black dashed line) plotted together with [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: One-sided p-value for the null hypothesis that the Area Under the Curve (AUC) [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

  1. [1]

    Ardia, K

    Algaba, A., D. Ardia, K. Bluteau, S. Borms, and K. Boudt (2020). Econometrics meets senti- ment: An overview of methodology and applications. Journal of Economic Surveys 34 (3), 512–547

    work page 2020

  2. [2]

    Baker, J

    Altig, D., S. Baker, J. M. Barrero, N. Bloom, P. Bunn, S. Chen, S. J. Davis, J. Leather, B. Meyer, E. Mihaylov, et al. (2020). Economic uncertainty before and during the COVID- 19 pandemic. Journal of Public Economics 191 , 104274

    work page 2020

  3. [3]

    Armesto, M. T., R. Hern´ andez-Murillo, M. T. Owyang, and J. Piger (2009). Measuring the information content of the Beige Book: A mixed data sampling approach. Journal of

    work page 2009

  4. [4]

    Aruoba, S. B., F. X. Diebold, and C. Scotti (2009). Real-time measurement of business conditions. Journal of Business & Economic Statistics 27 (4), 417–427

    work page 2009

  5. [5]

    Baker, S. R., N. Bloom, and S. J. Davis (2016). Measuring economic policy uncertainty. Quarterly Journal of Economics 131 (4), 1593–1636

    work page 2016

  6. [6]

    Balke, N. S. and D. Petersen (2002). How well does the Beige Book reflect economic activity? evaluating qualitative information quantitatively. Journal of Money, Credit and Banking , 114–136

    work page 2002

  7. [7]

    Consoli, and S

    Barbaglia, L., S. Consoli, and S. Manzan (2023). Forecasting with economic news. Journal of Business & Economic Statistics 41 , 708–719

    work page 2023

  8. [8]

    Consoli, and S

    Barbaglia, L., S. Consoli, and S. Manzan (2024). Forecasting GDP in Europe with textual data. Journal of Applied Econometrics 39 (2), 338–355

    work page 2024

Show all 30 references
  1. [9]

    Sentiment Analysis of Economic Text: A Lexicon-Based Approach

    Barbaglia, L., S. Consoli, S. Manzan, L. Tiozzo Pezzoli, and E. Tosetti (2024). ECIN Repli- cation Package for “Sentiment Analysis of Economic Text: A Lexicon-Based Approach”. Ann Arbor, MI: Inter-university Consortium for Political and Social Research

    2024

  2. [10]

    Gnabo, and G

    Bernal, O., J.-Y. Gnabo, and G. Guilmin (2016). Economic policy uncertainty and risk spillover in the Eurozone. Journal of International Money and Finance 65 (C), 24–45. 34

    2016

  3. [11]

    Garud, J

    Correa, R., K. Garud, J. M. Londono, and N. Mislang (2021). Sentiment in central banks’ financial stability reports. Review of Finance 25 (1), 85–120

    2021

  4. [12]

    Curtin, R. T. (1982). Indicators of consumer behavior: The University of Michigan surveys of consumers. Public Opinion Quarterly 46 (3), 340–352. De Marneffe, M.-C., T. Dozat, N. Silveira, K. Haverinen, F. Ginter, J. Nivre, and C. D. Manning (2014). Universal Stanford dependen...

    1982

  5. [13]

    Da, and P

    Engelberg, J., Z. Da, and P. Gao (2015). The sum of all FEARS investor sentiment and asset prices. Review of Financial Studies 28 , 1–32

    2015

  6. [14]

    Garcia, D. (2013). Sentiment during recessions. The Journal of Finance 68 (3), 1267–1300

    2013

  7. [15]

    Scotti, and C

    Gardner, B., C. Scotti, and C. Vega (2022). Words speak as loudly as actions: Central bank communication and the response of equity prices to macroeconomic announcements. Journal of Econometrics 231 (2), 387–409

    2022

  8. [16]

    Kelly, and M

    Gentzkow, M., B. Kelly, and M. Taddy (2019). Text as data. Journal of Economic Litera- ture 57 (3), 535–74

    2019

  9. [17]

    Hansen, S. and M. McMahon (2016). Shocking language: Understanding the macroeconomic effects of central bank communication. Journal of International Economics 99 , S114–S133

    2016

  10. [18]

    McMahon, and A

    Hansen, S., M. McMahon, and A. Prat (2018). Transparency and deliberation within the FOMC: a computational linguistics approach. The Quarterly Journal of Eco- nomics 133 (2), 801–870

    2018

  11. [19]

    Hassan, T. A., S. Hollander, L. Van Lent, and A. Tahoun (2019). Firm-level political risk: Measurement and effects. The Quarterly Journal of Economics 134 (4), 2135–2202

    2019

  12. [20]

    Hutto, C. and E. Gilbert (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media , Volume 8, pp. 216–225. 35

    2014

  13. [21]

    Larsen, V. H., L. A. Thorsrud, and J. Zhulanova (2021). News-driven inflation expectations and information rigidities. Journal of Monetary Economics 117 , 507–520

    2021

  14. [22]

    Loughran, T. and B. McDonald (2011). When is a liability not a liability? Textual analysis, dictionaries and 10-Ks. Journal of Finance 66 (1), 35–65

    2011

  15. [23]

    Petracco-Giudici, and M

    Nardo, M., M. Petracco-Giudici, and M. Naltsidis (2016). Walking down Wall Street with a tablet: A survey of stock market predictions using the web. Journal of Economic Surveys 30 (2), 356–369

    2016

  16. [24]

    Pang, B. and L. Lee (2008). Opinion mining and sentiment analysis. Foundations and Trends in information retrieval 2 (1–2), 1–135

    2008

  17. [25]

    Renault, T. (2017). Intraday online investor sentiment and return patterns in the US stock market. Journal of Banking & Finance 84 , 25–40

    2017

  18. [26]

    Shapiro, A. H., M. Sudhof, and D. J. Wilson (2022). Measuring news sentiment. Journal of Econometrics 228 , 221–243

    2022

  19. [27]

    Shapiro, A. H. and D. J. Wilson (2022). Taking the Fed at its word: A new approach to esti- mating central bank objectives using text analysis.The Review of Economic Studies 89 (5), 2768–2805. Ter Ellen, S., V. H. Larsen, and L. A. Thorsrud (2021). Narrative monetary policy su...

    2022

  20. [28]

    Tetlock, P. C. (2007). Giving content to investor sentiment: the role of media in the stock market. Journal of Finance 62 (3), 1139–1168

    2007

  21. [29]

    Tetlock, P. C., M. Saar-Tsechansky, and S. Macskassy (2008). More than words: quantify language to measure firms’ fundamentals. Journal of Finance 63 (3), 1437–1467

    2008

  22. [30]

    Thorsrud, L. A. (2018). Words are the new numbers: A news coincident index of the business cycle. Journal of Business & Economic Statistics , 1–17. 36

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.