REVIEW 4 major objections 4 minor 43 references
Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read GPT-4 recovers 77% of literary metaphor analogy frames
desk verdict A useful, honest small benchmark for metaphoric analogy extraction, with a real but manageable annotation-reliability caveat and one overreach in the conclusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the four-slot analogy frame $T_1:T_2::S_1:S_2$, which the task forces the model to fill by outputting one noun phrase per slot, with $T_1$ and $T_2$ in the target domain (the topic) and $S_1$ and $S_2$ in the source domain (the metaphoric image). The dataset supplies gold frames for 204 sourced literary metaphors, about half with at least one implicit slot. The evaluation counts an answer correct when the lemmatized head noun of the model's phrase matches the gold concept, ignoring modifier variation and plural/singular differences. The single-step prompt provides seven in-context examples and asks the model to complete the remaining slots, testing raw analogical extraction without fine-tuning.
What would settle it
Have a panel of fresh experts independently label a sample of the same texts, then compare the model-extracted quadruples to the panel's majority vote; if model agreement with the new panel is no better than the original annotators' agreement with each other, the reported accuracies would reflect task ambiguity rather than reliable extraction of structure.
Extended reading notes
Core claim
Given a short literary text containing a metaphor and one of its four concept slots, the paper claims that recent instructed LLMs can identify the three remaining concepts forming the proportional analogy $T_1:T_2::S_1:S_2$, distinguishing target-domain from source-domain terms. On the new 204-instance dataset, GPT-4 achieves 0.77 frame-wise accuracy and 0.61 full-quadruple accuracy under lemmatized head-noun matching, with Mixtral 8*22 close behind at 0.75 and 0.58. When a slot is implicit in the text, GPT-4 generates a concept that human raters score on average 1.21 out of 2, meaning relevant but imperfect inference. Accuracy stays stable as the noun count in the text rises, which the paper reads as a sign the method can extend to longer passages.
Load-bearing premise
The assumption that the gold labels, produced mostly by one expert and double-checked by a second author, are correct enough to score model outputs, even though the best pairwise annotator agreement on a complete quadruple is only a Cohen's kappa of 0.54.
Editorial extensions
If this is right
- The single-step prompting approach can convert any short text containing a metaphor into a structured four-term mapping in one model call, without domain-specific training.
- Because accuracy does not decline as the number of nouns in the sentence increases, the same method is a candidate for scaling to longer texts such as paragraphs or pages.
- The released 204-instance dataset and its evaluation protocol give future work a benchmark for metaphor extraction that goes beyond single-pair tagging.
- The gap between frame-blind and frame-wise scores suggests a two-step pipeline (extract terms, then assign frames) could close part of the remaining error, a direction the paper explicitly suggests.
Reading between the lines
- A likely next step is full open extraction, where the model must also locate the anchor concept itself rather than being given one slot; the paper's experimental design does not test this, but the high frame-wise scores suggest it could work on curated short texts.
- The weak inter-annotator agreement on complete quadruples (kappa 0.28-0.54) implies that the 0.61 full-quadruple accuracy for GPT-4 may be near the level of agreement among competent humans, so absolute comparisons to human agreement would be a more informative benchmark than raw accuracy.
- The same prompting recipe could be tried on non-literary metaphor-rich text such as news headlines or social media posts, where surface forms are less curated; the paper's finding that accuracy is robust to noun count gives some reason to expect transfer.
- The dataset deliberately excludes nested and multi-relation metaphors, so the current result covers only single-relation proportional analogies; a broader claim about metaphor extraction in general would require extending the frame to triples or multi-relation mappings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces a new task and dataset for extracting four-term metaphoric analogies of the form T1:T2::S1:S2 from short literary texts. The dataset contains 204 sourced instances labeled by an expert annotator (A5) and double-checked by a second author, with a 20-instance inter-annotator agreement study. The authors evaluate several out-of-the-box instructed LLMs (GPT-3.5, GPT-4, Llama-3-70B, Mixtral 8x7B/8x22B) in a few-shot prompting setup, measuring both extraction of the explicit frames and generation of implicit terms. The central results are that GPT-4 achieves the highest frame-wise accuracy (0.77 lemmatized head-noun match) and full-quadruple accuracy (0.61), and that GPT-4's generated implicit terms receive an average human relevance score of 1.21 out of 2, outperforming Mixtral 8x22B. The paper also analyzes performance by input/output frame, number of implicit terms, and sentence length.
Significance. If the main results hold, this is a useful contribution: it introduces a novel, publicly released benchmark for a task that has not been systematically evaluated before, and it provides evidence that large language models can recover the structure of proportional metaphoric analogies from natural text without task-specific training, including inferring implicit terms. The evaluation protocol is a clean, independent prompt-and-measure setup with no fitted parameters, and the authors release the dataset and scripts, which supports reproducibility. However, the load-bearing assumptions about the quality of the gold standard and the validity of the automatic metric need to be addressed before the quantitative claims can be taken at face value.
major comments (4)
- The headline full-quadruple accuracies (e.g., GPT-4 Q=0.61 in Table 6) are computed against gold labels produced almost entirely by one expert annotator (A5) on 204 instances, with only 20 instances independently annotated for IAA. The pairwise Cohen's kappa on complete quadruples (column Q in Table 2) is weak, ranging from 0.28 to 0.54 (average ~0.40). The authors acknowledge this in the Limitations section, but the central claim of the paper rests on these numbers. To make the claim robust, the manuscript should either provide multi-annotator validation for a larger portion of the dataset or report model accuracies against alternative gold labels (e.g., the majority vote or adjudicated labels on the IAA set) to show that the relative model rankings and the GPT-4 Q score are stable under reasonable annotator variation. As it stands, the Q=0.61 figure may reflect agreement with one particular interpretation rather than a robust structural ability.
- All accuracy numbers use the lemmatized head-noun matching metric, but the paper states explicitly that no manual evaluation of analogy frame extraction was performed. The authors describe the metric as a 'lower bound' on true accuracy, but no evidence is provided to support this direction or to quantify the gap. This is a load-bearing issue because the model comparisons in Tables 4 and 6 rely entirely on this automatic metric. The manuscript should include a human evaluation of a sample of model extractions (for example, the same 50-sentence protocol used for implicit-term evaluation) to estimate the metric's precision and recall relative to human judgment, and to confirm that the ranking of models is not an artifact of the matching function.
- The comparison between GPT-4 and Mixtral 8x22B is presented as a close call (0.77 vs 0.75 frame-wise; 0.61 vs 0.58 Q), but no statistical significance testing or confidence intervals are reported. Only GPT-3.5 was run on 10 batches (std=0.03); the other models were averaged over only 3 batches. The claim that GPT-4 is the best model, and the more general ordering of models, would be more credible if the authors reported variance across batches or instances, or ran a paired significance test. This is specific to the comparison in Table 4 and Table 6.
- The conclusion states that the performance of the models is 'remarkably high, in line with human annotators.' This comparison is not supported by the reported metrics: human agreement is measured as Cohen's kappa (chance-corrected) in Table 2, while model performance is reported as raw accuracy in Tables 4 and 6. These numbers are not directly comparable. To support the claim, the authors should compute a model-human agreement measure on a common set of instances (e.g., the 20-instance IAA set) or rephrase the claim to avoid implying direct comparability.
minor comments (4)
- The phrase 'even including the frames that are have implicit values' should read 'even including the frames that have implicit values.'
- The caption reads 'with on example sentence'; this should be 'with one example sentence.'
- The column header 'In Acc. for fields' is unclear; consider using 'Input' and 'Frame accuracy' explicitly.
- The publisher name 'John Hopking University Press' should be 'Johns Hopkins University Press.'
Circularity Check
No circularity: the evaluation compares LLM outputs to independently constructed gold annotations; no prediction reduces to its own inputs.
full rationale
The paper makes no derivation claim that could reduce to its own inputs. Its central empirical result—GPT-4 reaching 0.77 frame-wise accuracy and 0.61 full-quadruple accuracy—is obtained by prompting models with texts and a provided concept, then comparing head-noun matches against gold quadruples produced by expert annotation. No parameter is fitted to the evaluation data, and no model output is used to construct the gold labels. The in-context examples are sampled from the same dataset, but the paper states that 'the seven examples used for in-context learning input are removed from the test set reducing it to 197 instances,' so the evaluation is not circular. The main limitations disclosed in the paper—single-annotator labels for most instances, weak quadruple-level inter-annotator agreement, and the possibility that famous literary metaphors were seen by LLMs at pretraining—are concerns about benchmark validity and score stability, not about circularity. Self-citations such as Ushio et al. (2021) support a background claim about distant analogies and are not load-bearing for the reported results. Therefore, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (2)
- Number of in-context examples (prompt design) =
7
- Number of evaluation batches used for model comparison =
3 (10 for GPT-3.5)
assumptions (4)
- domain assumption All annotated texts can be decomposed into exactly one proportional analogy of the form T1:T2::S1:S2.
- domain assumption Metaphors are treated as a subset of analogies, following Gentner and Bowdle.
- domain assumption The expert annotator A5's labels, plus majority vote on a small IAA set, constitute reliable ground truth.
- domain assumption The lemmatized head-noun match is an adequate proxy for correctness of extracted phrases.
Cite this review
Pith. "Pith review of Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation." pith.science (2026). https://pith.science/paper/5TKEWVEO
@misc{pith2026241215375,
author = {Pith},
title = {Pith review of: Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/5TKEWVEO}},
note = {Machine review of arXiv:2412.15375}
}
read the original abstract
Extracting metaphors and analogies from free text requires high-level reasoning abilities such as abstraction and language understanding. Our study focuses on the extraction of the concepts that form metaphoric analogies in literary texts. To this end, we construct a novel dataset in this domain with the help of domain experts. We compare the out-of-the-box ability of recent large language models (LLMs) to structure metaphoric mappings from fragments of texts containing proportional analogies. The models are further evaluated on the generation of implicit elements of the analogy, which are indirectly suggested in the texts and inferred by human readers. The competitive results obtained by LLMs in our experiments are encouraging and open up new avenues such as automatically extracting analogies and metaphors from text instead of investing resources in domain experts to manually label data.
Figures
Reference graph
Works this paper leans on
-
[1]
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. https://doi.org/10.3115/980845.980860 The B erkeley F rame N et project . In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86--90, Montreal, Quebec, Canada. Association for Computational Linguistics
arXiv 1998
-
[2]
Robert Baldwin and Ruth Paris. 1982. The book of similes. Routledge & Kegan Paul
work page 1982
-
[3]
Zou, Venkatesh Saligrama, and Adam Kalai
Tolga Bolukbasi, Kai - Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Kalai. 2016. https://arxiv.org/abs/1607.06520 Man is to computer programmer as woman is to homemaker? debiasing word embeddings . CoRR, abs/1607.06520
arXiv 2016
-
[4]
Brian Bowdle and Dedre Gentner. 2005. https://doi.org/10.1037/0033-295X.112.1.193 The career of metaphor . Psychological review, 112:193--216
-
[5]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...
work page 2020
-
[6]
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. https://arxiv.org/abs/2303.12712 Sparks of artificial general intelligence: Early experiments with gpt-4 . Preprint, arXiv:2303.12712
arXiv 2023
-
[7]
Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.153 Scientific and creative analogies in pretrained language models . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2094--2100, Abu Dhabi, United Arab Emirates. Association for Computational ...
-
[8]
Mengshi Ge, Rui Mao, and Erik Cambria. 2022 a . https://doi.org/10.1609/aaai.v36i10.21313 Explainable metaphor identification inspired by conceptual metaphor theory . Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10681--10689
Show all 43 references
-
[9]
Mengshi Ge, Rui Mao, and Erik Cambria. 2022 b . https://doi.org/10.21203/rs.3.rs-2220808/v1 A survey on computational metaphor processing techniques: From identification, interpretation, generation to application
2022 doi
-
[10]
Dedre Gentner, Brian Bowdle, Phillip Wolff, and Consuelo Boronat. 2001. Metaphor is like analogy. Metaphor Is Like Analogy
2001
-
[11]
Joseph Grady. 1999. https://www.jbe-platform.com/content/books/9789027284013-cilt.175.06gra A typology of motivation for conceptual metaphor: correlation vs. resemblance . In Metaphor in Cognitive Linguistics. John Benjamins
1999
-
[12]
Mardy Grothe. 2008. I never metaphor I didn't like : a comprehensive compilation of history's greatest analogies, metaphors, and similes. HarperCollins
2008
-
[13]
Hofstadter and E
D.R. Hofstadter and E. Sander. 2013. https://books.google.fr/books?id=XkQT5eTnurYC Surfaces and Essences: Analogy as the Fuel and Fire of Thinking . Basic Books
2013
-
[14]
Xiaoyang Hu, Shane Storks, Richard Lewis, and Joyce Chai. 2023. https://doi.org/10.18653/v1/2023.acl-long.109 In-context analogical reasoning with pre-trained language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1...
2023 doi
-
[15]
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...
2024 arXiv
-
[16]
Xiaotong Jiang, Qingqing Zhao, Yunfei Long, and Zhongqing Wang. 2022. https://doi.org/10.18653/v1/2022.findings-acl.306 C hinese synesthesia detection: New dataset and models . In Findings of the Association for Computational Linguistics: ACL 2022, pages 3877--3887, Dublin, Ir...
2022 doi
-
[17]
Albert Katz, Allan Paivio, Marc Marschark, and Jim Clark. 1988. https://doi.org/10.1207/s15327868ms0304_1 Norms for 204 literary and 260 nonliterary metaphors on 10 psychological dimensions . Metaphor and Symbol - METAPHOR SYMB, 3:191--214
1988 doi
-
[18]
George Lakoff and Mark Johnson. 1980. Metaphors we Live by. University of Chicago Press, Chicago
1980
-
[19]
Richard Landis and Gary G
J. Richard Landis and Gary G. Koch. 1977. http://www.jstor.org/stable/2529310 The measurement of observer agreement for categorical data . Biometrics, 33(1):159--174
1977
-
[20]
Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. https://arxiv.org/abs/2401.16012 Finding challenging metaphors that confuse pretrained language models . Preprint, arXiv:2401.16012
2024 arXiv
-
[21]
Zachary J. Mason. 2004. https://doi.org/10.1162/089120104773633376 C or M et: A computational, corpus-based conventional metaphor extraction system . Computational Linguistics, 30(1):23--44
2004 doi
-
[22]
Melanie Mitchell. 2021. https://arxiv.org/abs/2102.10717 Abstraction and analogy-making in artificial intelligence . CoRR, abs/2102.10717
2021 arXiv
-
[23]
Michael Mohler, Mary Brunson, Bryan Rink, and Marc Tomlinson. 2016. https://aclanthology.org/L16-1668 Introducing the LCC metaphor datasets . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16) , pages 4221--4227, Portoro z , S...
2016
-
[24]
Ortony. 1993. https://doi.org/10.1017/CBO9781139173865 Metaphor and Thought , 2 edition. Cambridge University Press
1993 doi
-
[25]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...
2022 arXiv
-
[26]
Brad Pasanek. 2015. Metaphors of Mind. An Eighteenth-Century Dictionary. John Hopking University Press
2015
-
[27]
Zhibin Peng and Omid Khatin-Zadeh. 2023. Research on metaphor processing during the past five decades: a bibliometric analysis. Humanities and Social Sciences Communications, 10(1):928
2023
-
[28]
Ekaterina Shutova, Lin Sun, and Anna Korhonen. 2010. https://aclanthology.org/C10-1113 Metaphor identification using verb and noun clustering . In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1002--1010, Beijing, China. Col...
2010
-
[29]
Zhivar Sourati, Filip Ilievski, Pia Sommerauer, and Yifan Jiang. 2024. https://doi.org/10.1162/tacl_a_00688 Arn: Analogical reasoning on narratives . Transactions of the Association for Computational Linguistics, 12:1063--1086
2024 doi
-
[30]
Sternberg, Roger Tourangeau, and Georgia Nigro
Robert J. Sternberg, Roger Tourangeau, and Georgia Nigro. 1993. Metaphor, induction, and social policy: The convergence of macroscopic and microscopic views, page 277–304. Cambridge University Press
1993
-
[31]
Stevenson, Mathilde ter Veen, Rochelle Choenni, Han L
Claire E. Stevenson, Mathilde ter Veen, Rochelle Choenni, Han L. J. van der Maas, and Ekaterina Shutova. 2023. https://arxiv.org/abs/2310.20384 Do large language models solve verbal analogies like children do? Preprint, arXiv:2310.20384
2023 arXiv
-
[32]
Kevin Stowe, Tuhin Chakrabarty, Nanyun Peng, Smaranda Muresan, and Iryna Gurevych. 2021. https://doi.org/10.18653/v1/2021.acl-long.524 Metaphor generation with conceptual mappings . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and ...
2021 doi
-
[33]
Oren Sultan and Dafna Shahaf. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.232 Life is a circus and we are the clowns: Automatically finding analogies between situations and processes . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process...
2022 doi
-
[34]
Xiaoyu Tong, Rochelle Choenni, Martha Lewis, and Ekaterina Shutova. 2024. https://arxiv.org/abs/2403.11810 Metaphor understanding challenge dataset for llms . Preprint, arXiv:2403.11810
2024 arXiv
-
[35]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...
2023 arXiv
-
[36]
Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Collados. 2021. https://doi.org/10.18653/v1/2021.acl-long.280 BERT is to NLP what A lex N et is to CV : Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Ass...
2021 doi
-
[37]
Tony Veale and Guofu Li. 2012. https://aclanthology.org/P12-3002 Specifying viewpoint and information need with affective metaphors: A system demonstration of the metaphor-magnet web app/service . In Proceedings of the ACL 2012 System Demonstrations , pages 7--12, Jeju Island,...
2012
-
[38]
Lennart Wachowiak and Dagmar Gromann. 2023. https://doi.org/10.18653/v1/2023.acl-long.58 Does GPT -3 grasp metaphors? identifying metaphor mappings with generative language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol...
2023 doi
-
[39]
Taylor Webb, Keith Holyoak, and Hongjing Lu. 2023. https://doi.org/10.1038/s41562-023-01659-w Emergent analogical reasoning in large language models . Nature Human Behaviour, 7:1--16
2023 doi
-
[40]
Shalin, Amitava Das, and Amit Sheth
Thilini Wijesiriwardene, Amit Sheth, Valerie L. Shalin, Amitava Das, and Amit Sheth. 2023. https://doi.org/10.1109/MIS.2023.3305862 Why do we need neurosymbolic ai to model pragmatic analogies? IEEE Intelligent Systems, 38(5):12–16
2023
-
[41]
Chi, and Denny Zhou
Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. 2024. https://arxiv.org/abs/2310.01714 Large language models as analogical reasoners . Preprint, arXiv:2310.01714
2024 arXiv
-
[42]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[43]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.