REVIEW 3 major objections 5 minor 20 references
Identifying economic narratives in large text corpora -- An integrated approach using Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that GPT-4o can extract economic narratives from news text in a structured format, but that the model falls short of expert-level performance on complex documents, reaching only 44% accuracy against a gold standard where…
desk verdict Solid, honestly reported benchmark of GPT-4o for economic narrative extraction, with a genuinely reusable codebook; the headline human–model gap is plausible but its magnitude is not fully trustworthy because the human baseline is scored against a gold standard built by the same three experts. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a two-part extraction procedure: an operational definition of an economic narrative as a causal connection between two temporally and semantically distinct events, expressed in the form 'A causes B' or 'A is caused by B'; and a few-shot chain-of-thought prompt for GPT-4o that guides the model through five discrete steps—focused excerpt, sequence of interest, causal restatement, coreference resolution, and event rephrasing—before checking for further narratives. The gold standard is built from consensus discussions among three expert annotators using a detailed codebook, and deviations are classified as major or minor. This machinery lets the authors compare model output and human deviations against a common benchmark and enables post-processing such as event decomposition, topic clustering, and valence assignment.
What would settle it
Re-annotate the 80 test documents with a fresh panel of experts who use only the written codebook and were not part of the consensus sessions, then compute their accuracy against the published gold standard. If their accuracy is similar to GPT-4o's (around 44%) rather than to the original annotators' 67–74%, the human-model gap is substantially an artifact of consensus construction; if fresh experts land at 67–74%, the gap reflects a genuine shortfall in the model.
Extended reading notes
Core claim
The paper's central claim is that a state-of-the-art instruction-tuned LLM, prompted with a concise codebook plus seven hand-coded input-output examples and an explicit chain-of-thought procedure, can extract economic narratives from newspaper excerpts in a structured, aggregable form, but that this capability falls short of expert-level performance. On the 80-document test set, GPT-4o achieved 44% accuracy against the gold standard, while the three expert annotators scored 72%, 74%, and 67%; the model produced 1.25 major deviations per document versus 0.35–0.49 for the experts. The model also shows a systematic bias in narrative density, averaging 2.32 narratives per document with a standard deviation of 1.15 versus 2.22–2.36 with standard deviations near 2 for humans, reflecting a tendency to over-identify narratives in sparse texts and under-identify in dense ones, particularly forked and chained causal structures. The authors interpret these results as evidence that LLMs are promising tools for scaling narrative research, but that expert judgment remains necessary for reliable annotation.
Load-bearing premise
The gold standard is produced through consensus sessions in which the same three experts who are later evaluated against it discuss and decide which narratives count, and the paper assumes this consensus yields an unbiased, valid measure of the true narratives.
Editorial extensions
If this is right
- LLM-based narrative extraction can be deployed as a scalable screening tool, but every output should be reviewed by a human coder for complex documents.
- Few-shot chain-of-thought prompting is an effective format for translating a human codebook into an LLM prompt for structured annotation tasks in economics.
- The model's tendency to compress forked or chained narratives into single statements means downstream counts of narrative density will be biased unless post-processing explicitly forks compound events.
- The aggregation pipeline of topic-valence clustering can turn raw extractions into recurring macro narrative arcs, such as 'loose monetary policy causes rising inflation'.
- Narrative extraction remains inherently subjective even among experts, so accuracy metrics should be interpreted relative to expert disagreement, not as absolute truth.
Reading between the lines
- If the consensus gold standard embeds the original annotators' interpretive preferences, the model's relative gap might narrow against a majority-vote or independently derived gold standard; this could be tested by comparing GPT-4o to a fresh panel's labels.
- The same prompting and evaluation approach could be applied to other economically salient topics—labor markets, housing, climate—where narrative density and structural complexity might differ.
- The observed inductive bias toward an average narrative count suggests that prompting strategies that explicitly vary the expected number of narratives (e.g., providing a per-document rarity prior or using a 'reject if none' option) could reduce both false positives in sparse texts and false negatives in dense ones.
- A testable extension is to measure whether adding decomposed fork/chain training signals in few-shots—rather than just instructions—improves the model's handling of complex causal structures.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops and evaluates a prompt-based LLM pipeline for extracting economic narratives from newspaper text. It defines an economic narrative as a causal link between two events, operationalizes this definition in a detailed codebook, builds a gold standard from three expert annotators on 100 WSJ/NYT excerpts, and compares GPT-4o, prompted with a few-shot Chain-of-Thought protocol, against that gold standard. The headline findings are that GPT-4o reaches 44% accuracy and 1.25 major deviations per document, while the three expert annotators reach 67–74% accuracy and 0.35–0.49 major deviations. The paper concludes that LLM-based extraction is feasible but not yet expert-level, and it recommends hybrid workflows in which human experts retain interpretive oversight.
Significance. If the central comparison were fully credible, this paper would be a useful methodological contribution: it provides a transferable codebook, a transparent prompt-engineering procedure, a publicly released annotation dataset, and a structured way to quantify narrative-extraction errors. The paper is also refreshingly honest about the subjectivity of narrative coding and about the exploratory nature of the aggregation step. Its main limitation is measurement: the human baseline is built by the same experts who are later scored against it, the accuracy metric is never defined, and no uncertainty quantification accompanies the headline numbers. These gaps affect the central claim that GPT-4o 'falls short of expert-level performance' and need to be addressed before the quantitative magnitude of the human–model gap can be taken at face value.
major comments (3)
- [Section 4.2, Table 2] The gold standard is constructed by the same three experts who are later evaluated against it: after individual coding, every narrative was discussed by all three annotators and included only upon unanimous agreement. The expert accuracy figures in Table 2 are therefore a measure of within-group consistency after deliberation, not an estimate of independent expert-level performance. Because GPT-4o had no access to the consensus discussions or the codebook workshops (Section 4.1), the human–model gap could be inflated by shared interpretive preferences that are embedded in the gold standard. The paper acknowledges that 'no narrative extraction can be truly objective' (Section 4.1), but it does not address this self-referential baseline. Please provide an external validation, for example by computing each expert's accuracy against a gold standard built from the other two experts' unanimous agreements, or by comparing the model against an independent annotator who was not involved in codebook development or consensus sessions.
- [Section 6.1, Table 2] The central quantitative claim rests on an 'accuracy' metric that is never defined. The reader cannot tell whether 44% and 67–74% refer to exact narrative-level matches, partial credit for near-miss narratives, token-level overlap, or some other rule. In addition, the paper reports only point estimates from a single run at temperature 0.2 on 80 documents, with no confidence intervals, significance tests, or multiple runs. The difference between 44% and 67% could plausibly be within sampling variation given such a small test set. Please define the accuracy metric precisely, report bootstrap confidence intervals or a significance test for the expert–model gap, and either run the model multiple times or use a temperature-0 protocol with multiple seeds to assess stability.
- [Section 5.2, Section 5.3] The final prompt and the number of few-shots were selected by evaluating 1–9 few-shots on a 20-document validation split, and the final few-shot set was then chosen by rotating combinations of seven examples. This is a legitimate model-selection procedure, but the reported test performance is conditional on a selection process with very few validation documents, so the expert–model gap could change under a different validation split. Please report the variability of test performance across few-shot sets or validate the chosen prompt on multiple random splits, and clarify explicitly that the 80 test documents were completely untouched during prompt selection.
minor comments (5)
- [Section 5.1] There is a typo in the sentence beginning 'Morover, since an LLM's conceptual understanding...'; it should read 'Moreover'.
- [Section 6.1] The sentence 'This translate to an unexpected major-deviation rate...' should read 'This translates to...'.
- [Section 5.3] The validation description is internally inconsistent: the paper first says 20 documents are used for cross-validation, then says the model is evaluated on the 'remaining 11 examples' when using 1–9 few-shots, and later states that performance peaks with 7 few-shots. With 20 validation documents and 7 few-shots, 13 evaluation examples would remain. Please clarify the exact split and evaluation protocol.
- [Table 2] The Jaccard similarity metric is described only as 'lexical overlap between predicted and reference token sets'; the tokenization scheme, whether events are compared separately or as full narratives, and how coreference-resolved variants are handled should be specified.
- [Section 3.2] The paper states that 100 documents were 'randomly sampled' from the corpus, but it does not report the sampling seed, any stratification, or the total corpus size. This information would help readers assess generalizability and reproducibility.
Circularity Check
Human baseline is scored against a gold standard built from the same experts' consensus, making the central human–model comparison partially self-referential.
-
self definitional
[Section 4.2 (Creating a gold-standard data set) and Table 2 in Section 6.1 (Narrative identification)]
"Each individually coded narrative was discussed by all three annotators. If they unanimously agreed it was correctly coded, the narrative was added to the list of gold-standard narratives for the corresponding document. ... The individual annotations by each annotator, however, were not discarded. Instead, we use them to compute an expected deviation from the gold-standard. ... Accuracy (vs. gold) 0.72 0.74 0.67 0.44"
The gold standard is the consensus of the three expert annotators, and those same annotators are then scored against it in Table 2. Each expert's accuracy is therefore agreement with a standard they themselves helped construct during the consensus sessions, and the 'expected deviation' baseline is the within-group spread around that self-authored standard. GPT-4o, by contrast, is an outside system with no access to those discussions, so the headline gap (44% vs. 67–74%) measures the model's alignment with the annotators' collective interpretive preferences rather than with an independently anchored truth. To the extent the consensus embedded those preferences, the human–model gap is inflated by construction.
full rationale
The paper's central result—that GPT-4o reaches 44% accuracy while expert annotators reach 67–74%—rests on a benchmark whose provenance is partially self-referential. The gold standard was created by consensus among the same three experts whose individual annotations are later scored against it as the human baseline. This does not make the model's accuracy a fitted or predicted quantity: the LLM was not trained on the test labels, and few-shot selection used only a 20-document validation split, with the remaining 80 documents held out. The model's 44% is an independent measurement of agreement with that gold standard. The circularity is limited to the definition of 'expert-level performance': the human baseline is not an external ground truth but an agreement-with-own-consensus measure, so the magnitude of the human–model gap is not fully separable from the gold standard's provenance. No load-bearing self-citations or imported uniqueness theorems were found; the citations to the authors' prior work are background and method-building references. The paper itself acknowledges in Section 4.1 that 'no narrative extraction can be truly objective,' which underscores that the gold standard is a constructed, interpretive benchmark. This is a genuine validity concern but not a derivation-level equivalence, so the overall circularity score is moderate rather than high.
Assumptions & free parameters
free parameters (4)
- Number of few-shot examples =
7
- Temperature =
0.2
- Excerpt window =
Filter sentence plus two preceding and two following sentences
- Topic anchor clusters =
Manually refined set of anchor terms and clusters
assumptions (5)
- domain assumption The consensus gold-standard process in Section 4.2 yields the true set of narratives for each document.
- domain assumption Expert annotators with near-native English and codebook training are the appropriate reference for narrative validity.
- domain assumption The codebook operationalization, narrative as two events connected by positive causality with explicit economic relevance, captures the theoretical concept of economic narratives.
- domain assumption GPT-4o outputs at temperature 0.2 are stable enough for a single-run evaluation.
- domain assumption The 100 randomly sampled documents are representative of WSJ/NYT inflation coverage from 1985 to 2023.
Cite this review
Pith. "Pith review of Identifying economic narratives in large text corpora -- An integrated approach using Large Language Models." pith.science (2026). https://pith.science/paper/UO2AWAIL
@misc{pith2026250615041,
author = {Pith},
title = {Pith review of: Identifying economic narratives in large text corpora -- An integrated approach using Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/UO2AWAIL}},
note = {Machine review of arXiv:2506.15041}
}
read the original abstract
As interest in economic narratives has grown in recent years, so has the number of pipelines dedicated to extracting such narratives from texts. Pipelines often employ a mix of state-of-the-art natural language processing techniques, such as BERT, to tackle this task. While effective on foundational linguistic operations essential for narrative extraction, such models lack the deeper semantic understanding required to distinguish extracting economic narratives from merely conducting classic tasks like Semantic Role Labeling. Instead of relying on complex model pipelines, we evaluate the benefits of Large Language Models (LLMs) by analyzing a corpus of Wall Street Journal and New York Times newspaper articles about inflation. We apply a rigorous narrative definition and compare GPT-4o outputs to gold-standard narratives produced by expert annotators. Our results suggests that GPT-4o is capable of extracting valid economic narratives in a structured format, but still falls short of expert-level performance when handling complex documents and narratives. Given the novelty of LLMs in economic research, we also provide guidance for future work in economics and the social sciences that employs LLMs to pursue similar objectives.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Basic Idea The goal of this coding task is to extract all narratives from the provided texts as accurately and comprehensively as possible. The texts are excerpts from newspaper articles published in the NYT or the Wall Street Journal.a Ruhr Economic Papers 43 We define a narrative as a causal connection between two consecutive events. Simple examples of ...
-
[2]
T arget F ormat We aim to train a language model to reduce all narratives in a text to one of two possible formats: • Format A: event 1 – causes – event 2 • Format B: event 1 – is caused by – event 2 Each narrative must follow one of these formats. The distinction between A and B preserves the order of events as they appear in the original text. The event...
work page 1912
-
[3]
the collapse of the Berlin Wall in 1989
Events 3.1 Types of Events Events can be categorized into events and activities as well as states and circumstances. • “the collapse of the Berlin Wall in 1989” (event) • “Mexico grew rapidly” (activity) • “Trump signed the bill into law” (activity) Ruhr Economic Papers 44 • “inflation” (state) • “lack of competitive innovation” (circumstance) States and ...
work page 1989
-
[4]
Only Positive Causal Links Please code only positive causal relationships. Ignore negative causality. Example: Putin invading Ukraine – does not cause – President Zelensky fleeing Ukraine (do not code) Ruhr Economic Papers 46
-
[5]
Please write each narrative on a new line
Multiple Narratives per T ext A document may contain multiple narratives. Please write each narrative on a new line. Do not use enumerations. Correct: - the prices for cucumbers rose by 100% this month – causes – people stop buying cucumbers - huge money supply – causes – prices for real estate go up Wrong:
-
[6]
You can ignore all other narratives
Thematic Scope 6.1 Explicit Economic Reference Please only code narratives that have an explicit economic relevance. You can ignore all other narratives. If you are unsure, write down the narrative and add a note (uncertain). Examples: • Putin invading Ukraine – causes – rising energy prices (code) • Putin invading Ukraine – causes – enormous military sup...
-
[7]
Preserve Content This point is similar to point 3.2. Sometimes, there is a strong temptation to omit parts of a sentence in order to make the narrative more pointed. In some Ruhr Economic Papers 47 cases, that’s acceptable; however, the rule should be to include all parts of the sentence that are necessary to preserve the core message of the narrative. Pl...
-
[8]
the prices for cucumbers rose ... | 2. huge money supply
Show all 20 references
-
[11]
Semi"-causal cues (such as
Edge Cases If you are unsure whether something really constitutes a narrative or not, code it anyway to be on the safe side. You can later compare all such cases with the other coders or with us. Example 1: "Semi"-causal cues (such as "as", "suggesting", ...) can be interprete...
2004
-
[12]
Fork: If Event A is said to be a common cause of Events B and C, code both pathways as separate narratives
-
[13]
Ruhr Economic Papers 52 ## Rules: You also have to follow a couple of hard-and-fast rules:
Chain: If Event B is said to be a mediator between Event A and Event C, also code both pathways as separate narratives. Ruhr Economic Papers 52 ## Rules: You also have to follow a couple of hard-and-fast rules:
-
[14]
Event A", the event that appears second must be coded as
Retain the order of the two events from the source text at all times. The event that appears first in the text must be coded as "Event A", the event that appears second must be coded as "Event B"
-
[15]
Target Form
When you state the narrative in its "Target Form", always represent the causal connection by using one of the following phrasings: i) "causes": Use this causal connector when Event A is the cause and Event B is an effect of Event A. ii) "is caused by": Use this causal connecto...
-
[16]
Focused Excerpt
"Focused Excerpt": The excerpts vary in length and in their narrative density. Parts of each excerpt may obviously not contain any narrative. To focus on relevant parts of the excerpt, start by repeating it, but leaving out the sentences that you are sure no narrative appears in
-
[17]
Sequence of Interest
"Sequence of Interest": Go through the "Focused Excerpt" and state the first narrative sequence you find, that is a sequence that contains two events and what you consider to be a causal connection between them
-
[18]
Causal Restatement
"Causal Restatement": Based on the current "Sequence of Interest", restate the narrative in the appropriate target form. Make sure to repeat the events verbatim without rephrasing
-
[19]
Coreference Resolution
"Coreference Resolution": If necessary, rephrase one or both events to clearly identify the entities that occur in the narrative. For the most part, this will involve replacing personal pronouns with the entity itself
-
[20]
Event Rephrasing
"Event Rephrasing": If necessary, rephrase one or both events again so that the narrative uses correct language. After every narrative, continue by circling back to 1) and restate the "Focused Excerpt". Do so even if you did not previously detect more narratives in it. Use it ...
-
[53]
Lex2Sent: A bagging ap- proach to unsupervised sentiment analysis
url: https://old.gscl.org/media/pages/arbeitskreise/cpss/cpss-2022/ workshop- proceedings- 2022/254133848- 1662996909/cpss- 2022- proceedings. pdf. Lange, Kai-Robin, Jonas Rieger, and Carsten Jentsch (2024). “Lex2Sent: A bagging ap- proach to unsupervised sentiment analysis”. ...
2024
-
[2021]
Dy- namicchangedetectionintopicsbasedonrollingLDAs
Association for Computational Linguistics, pp. 2337–2347.doi: 10.18653/v1/ 2021.findings-emnlp.201. Rieger, Jonas, Kai-Robin Lange, Jonathan Flossdorf, and Carsten Jentsch (2022). “Dy- namicchangedetectionintopicsbasedonrollingLDAs”.In: Proceedings of the Text2Story’22 Worksho...
2022 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.