It's not Rocket Science : Interpreting Figurative Language in Narratives

Tuhin Chakrabarty; Vered Shwartz; Yejin Choi

RECORD

It's not Rocket Science : Interpreting Figurative Language in Narratives

Tuhin Chakrabarty, Yejin Choi, Vered Shwartz

Computation and Language · Submitted 31 Aug 2021

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim The knowledge-enhanced models improve the performance on both the discriminative and generative tasks, further bridging the gap from human performance.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2109.00087 (version 3)

Pith IDVES7HKIP, Pith's permanent record of this paper and its review

CitationTuhin Chakrabarty, Vered Shwartz, Yejin Choi. "It's not Rocket Science : Interpreting Figurative Language in Narratives." arXiv:2109.00087 (2021). https://pith.science/paper/VES7HKIP

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,717 input tokens.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.51 · human participants 0.85 · controlled comparison 0.84 · randomized assignment 0.02 · states quantitative result 0.08 · compares with prior results 0.92 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v3. Results and the main claim are underlined; show every role.

Figurative language is ubiquitous in English. Yet, the vast majority of NLP research focuses on literal language. Existing text representations by design rely on compositionality, while figurative language is often non-compositional. In this paper, we study the interpretation of two non-compositional figurative languages (idioms and similes). We collected datasets of fictional narratives containing a figurative expression along with crowd-sourced plausible and implausible continuations relying on the correct interpretation of the expression. We then trained models to choose or generate the plausible continuation. Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks. We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language types : inferring meaning from the context and relying on the constituent words' literal meanings. The knowledge-enhanced models improve the performance on both the discriminative and generative tasks, further bridging the gap from human performance.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Figurative language is ubiquitous in English.	background 1.00	0.04	0.01	asserted 1.00
s1	Yet, the vast majority of NLP research focuses on literal language.	gap 0.64	0.07	0.02	asserted 1.00
s2	Existing text representations by design rely on compositionality, while figurative language is often non-compositional.	background 0.64	0.05	0.03	asserted 0.89
s3	In this paper, we study the interpretation of two non-compositional figurative languages (idioms and similes).	contribution 1.00	0.70	0.03	asserted 0.96
s4	We collected datasets of fictional narratives containing a figurative expression along with crowd-sourced plausible and implausible continuations relying on the correct interpretation of the expression.	method 0.98	0.97	0.02	asserted 0.99
s5	We then trained models to choose or generate the plausible continuation.	method 0.99	0.95	0.03	asserted 0.92
s6	Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks.	result 1.00	0.95	0.29	asserted 1.00
s7	We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language types : inferring meaning from the context and relying on the constituent words' literal meanings.	method 0.80	0.88	0.13	asserted 0.95
s8	The knowledge-enhanced models improve the performance on both the discriminative and generative tasks, further bridging the gap from human performance.	result 0.99	0.94	0.58	asserted 1.00

What kind of paper the classifier reports this is: empirical study 0.65 · evidence from observation or experiment 0.99.

What was examined, and what was not

sources read

arXiv listing abstract v3

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Figurative language is ubiquitous in English. Yet, the vast majority of NLP research focuses on literal language. Existing text representations by design rely on compositionality, while figurative language is often non-compositional. In this paper, we study the interpretation of two non-compositional figurative languages (idioms and similes). We collected datasets of fictional narratives containing a figurative expression along with crowd-sourced plausible and implausible continuations relying on the correct interpretation of the expression. We then trained models to choose or generate the plausible continuation. Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks. We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language types : inferring meaning from the context and relying on the constituent words' literal meanings. The knowledge-enhanced models improve the performance on both the discriminative and generative tasks, further bridging the gap from human performance.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

7 arXiv papers cite this in 14 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 5 sentences without a confident label

Uses its data or values 1 paper

Specifically, we use the Figurative Narrative Benchmark (FNB) dataset [ref] , where continuation selection requires correct IE interpretation.

IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions · Tasks in the presence of IEs · 0.98

Compares with it 1 paper

Note that while state-of-the-art is around $83$%, its method is much more involved, as it includes both generation and classification, and is followed by what is essentially a $12$ inference ensemble for each sample [ref] .

IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions · Application Results · 0.93

Cites it as support 1 paper

For example, while PLMs exhibit a certain ability to interpret similes after fine-tuning [ref] , they can hardly understand similes that require common-sense knowledge [ref] .

MAPS-KB: A Million-scale Probabilistic Simile Knowledge Base · Introduction · 0.99

Cites it as background 5 papers

There is also a bulk of works designing specific tasks to investigate whether PLMs have the ability to interpret similes, such as simile property probing task [ref] , Winograd-style simile understanding [ref] and simile continuation selection and generation [ref] .

MAPS-KB: A Million-scale Probabilistic Simile Knowledge Base · Simile Processing Tasks · 0.96
[ref] interpret similes by choosing or generating continuation for narratives via PLMs.

Can Pre-trained Language Models Interpret Similes as Smart as Human? · Related Work · 0.99
[ref] collected datasets on idiom and simile and developed knowledge enhanced RoBERTa-based models.

Figuratively Speaking: Authorship Attribution via Multi-Task Figurative Language Modeling · Related Work · 0.84

What it cites · 1 sentence without a confident label

Uses the method or tool of 2 papers

We use GPT-2 XL [ref] and GPT-3 [ref] to compute the normalized log-likelihood score of each continuation given the narrative, predicting the continuation with the highest probability: $argmax_i P_LM(C_i |N)$.

Language Models are Few-Shot Learners · Methods · 1.00
We fine-tune RoBERTa-large [ref] as a multiple-choice model.

RoBERTa: A Robustly Optimized BERT Pretraining Approach · Methods · 1.00

Cites as background 1 paper

Since the release of this dataset, many computational approaches for the task were developed [ref] .

Language Models are Few-Shot Learners · Narrative Understanding · 0.89

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.