REVIEW 4 major objections 4 minor 47 references
Enhancing Rhetorical Figure Annotation: An Ontology-Based Web Application with RAG Integration
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that combining a reified rhetorical-figure ontology with retrieval-augmented generation lets untrained users annotate German rhetorical figures, with basic chunking (size 2048, top-12 reranked to top-6) as the best RAG…
desk verdict A genuine engineering contribution—reified GRhOOT ontology, a German annotation web app, and a RAG pipeline—whose evaluation is too weak to support the claims as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The reified GRhOOT ontology is the central object. In the original ontology, a figure like epiphora was described by compound relations such as 'isRepeatableElementOfSameForm'; reification splits these into three fine-grained properties, hasOperation, affectedElement, and operationalForm, on which users can filter. This decomposition is what lets a non-expert describe a figure's pattern instead of knowing its name. The second load-bearing mechanism is the RAG pipeline: the ontology is chunked and embedded into a vector store, a user question is embedded and matched against chunks, the top chunks are reranked, and a language model answers with that retrieved context. The paper evaluates the pipeline using ontology competency questions and an automated RAG evaluation suite, comparing chunk sizes, chunking methods, and retrieval depths to select the best setting.
What would settle it
Give the application to a group of untrained, non-expert German speakers with a set of sentences that contain figures based on transferred meaning, such as metaphors, and measure how often their property selections lead to the correct figure name. A success rate near chance for those figures would show that the property-based flow does not support untrained annotation for the very class of figures the paper admits are hardest.
Extended reading notes
Core claim
The paper's central claim is that an ontology of rhetorical figures can be made practically useful for annotation by reifying its relations into user-facing property dimensions and by grounding a language-model chat in retrieval from that ontology. A user who spots a pattern, such as repetition of a word at the same position, selects the matching properties from dropdowns; the app translates the selection into a SPARQL query and returns candidate figures with definitions and examples. The same reified ontology serves as the retrieval source for a RAG-powered chat, and the paper reports that a basic chunking setting (chunk size 2048, retrieve top-12 then rerank to top-6) yields the best answer-oriented metrics among the configurations tested. The approach is described as one of the first to combine a rhetorical ontology with RAG for practical annotation, and the paper claims it shows promising results, while noting that figures based on transferred meaning, such as metaphor, remain harder for this property-selection flow.
Load-bearing premise
The entire annotation flow assumes that a user without linguistic training can reliably map a text to the three property dimensions of operation, affected element, and operational form; if users cannot do that, the app returns the wrong candidates and the collected data lose quality.
Editorial extensions
If this is right
- Users without linguistic training can submit German examples and receive candidate figure names with definitions, which lowers the barrier to creating annotated data.
- Because figures can be queried by operation, affected element, or operational form, the same ontology supports flexible search paths rather than requiring users to know figure names in advance.
- The RAG configuration identified in the paper, basic chunking with a chunk size of 2048 and top-12 to top-6 reranking, can serve as a default starting point for other ontology-based RAG applications.
- Questions that require aggregating information across multiple ontology chunks (for example, listing all figures with a given property) remain a weak spot for the RAG chat, so the app is better suited to direct look-ups.
- Once user-submitted examples are verified, they can be added back to the ontology and the vector store, giving a concrete route to grow the sparse German rhetorical-figure data.
Reading between the lines
- The same reification pattern could transfer to other low-resource annotation domains: any domain ontology whose compound relations can be split into user-facing property dimensions could power a similar property-selection interface.
- The RAG result was tuned on a small, compact ontology; larger or more heterogeneous knowledge bases might not favor basic chunking at 2048, so the optimal setting is likely scale-dependent.
- Because the LLM's wordy answers and modern paraphrasing lowered semantic-similarity scores, future evaluations should separate factual correctness from phrasing, for example by also scoring exact figure-name matches.
- A head-to-head comparison of the dropdown property flow against the RAG chat flow, using the same sentences and annotators, would show which interface yields more accurate annotations and where the 'No idea' fallback is needed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents "Find your Figure," a Flask-based web application for annotating rhetorical figures in German, built on a reified version of the GRhOOT ontology. Users select properties such as operation, affected element, and operational form from dropdowns; the backend translates these into a SPARQL query and returns candidate figures. A separate chat page integrates a retrieval-augmented generation (RAG) pipeline using gpt-3.5-turbo, bge-m3 embeddings, and a reranker. The authors evaluate several chunk sizes and retrieval settings with the Ragas framework on 70 competency questions derived from the ontology, and they conclude that basic chunking with chunk size 2048 and top-12 retrieval reranked to top-6 is the optimal configuration. They also describe input verification mechanisms, limitations for figures based on transferred meaning, and ethical considerations around intellectual property.
Significance. The paper addresses a real bottleneck in computational rhetoric: the scarcity of annotated data for non-English and non-metaphor rhetorical figures. The ontology restructuring, the openly available code, and the integration of RAG with a rhetorical ontology are useful contributions if the system works as described. The significance is currently limited, however, because the annotation workflow is not validated with users, and the RAG evaluation does not compare against a no-RAG baseline or provide error bars. The claim of identifying the optimal RAG configuration is therefore not yet supported, and the central data-collection goal rests on an untested assumption about untrained annotators.
major comments (4)
- [§5.1] The RAG evaluation is circular with respect to the ontology. The 70 competency questions are either template questions asking for properties of figures or questions generated by gpt-4 from formalizations of the same ontology; the ground-truth answers are obtained by running SPARQL queries on that ontology, and the context is manually extracted from that same ontology. Since the retrieval source and the answer source are identical, high context recall and high faithfulness partly measure whether the retriever finds the exact passage from which the answer was derived, not whether the pipeline produces correct answers on an independent task. Please either reframe the evaluation as a retrieval-coverage check on a single knowledge source or construct ground truth from an external corpus (e.g., dictionary definitions or expert-verified examples).
- [§5.2, Table 2] The choice of basic chunking 2048 with top-12/6 as the optimal setting is not consistent with the stated priority on answer metrics. In Table 2, this row has answer correctness 0.7355 and answer similarity 0.8655, while the AMR 512/256/128 top-6/3 row has answer correctness 0.8619 and the basic 2048 top-6/3 row has answer similarity 0.9673. If answer correctness and answer similarity are the most important metrics, as claimed at the end of §5.1, the selected configuration is not the best on those metrics. Please specify the exact selection criterion (e.g., a weighted combination or a threshold) and justify why the chosen setting is preferred over rows that score higher on answer correctness or answer similarity.
- [§4.1, §7] The core annotation workflow rests on an untested assumption: that users without linguistic knowledge can map a text onto the reified property dimensions (operation, affected element, operational form) and that the resulting SPARQL query reliably surfaces the intended rhetorical figure. The paper provides no user study, no inter-annotator agreement, and no pilot test of the FyF.html dropdown workflow. Section 7 concedes that the application is better suited to figures with obvious lexical patterns than to figures relying on transferred meaning, such as metaphors. That is not merely a user-side limitation; it directly limits the kind of data the application can collect. Please report at least a small pilot evaluation with untrained users, including agreement or accuracy against expert labels, or explicitly scope the application's data-collection claim to lexical-pattern figures.
- [§5] The evaluation compares only RAG configurations and never includes a no-RAG baseline, such as prompting the same LLM without retrieved ontology context or a retrieval-only baseline. Without such a baseline, the claim that RAG improves answer quality over a plain LLM is not supported. In addition, Table 2 reports a single run per configuration without standard deviations or repeated trials; even at temperature 0.1, LLM output is stochastic and the post-processing step (described in §5.1) could introduce run-to-run variation. Please add a no-RAG baseline and repeated runs with variance reporting, or explicitly weaken the conclusion to a comparison among the tested RAG settings rather than an identification of the globally optimal configuration.
minor comments (4)
- [§3, Table 1] The property name is inconsistent: Section 3 writes `:operationalForm :SameForm`, while Table 1 uses `hasOperationForm`. Please standardize the property names in the text, the table, and the ontology itself.
- [§5.1] The sentence "The answers of the LLM still require post-processing before we can use them in the Ragas framework" is clear, but the paper does not explain whether the same post-processing is applied consistently to all configurations or whether it could interact with the chunking settings. Please clarify in the evaluation description.
- [§4.2] The text says "we will flag the example in the database in the columnis_invalid," which appears to be a missing-space typo for `is_invalid`. Please fix the formatting.
- [§5.2] The phrase "We notice deviations in answer correctness and answer similarity ," contains an extra space before the comma. Please correct the typo.
Circularity Check
No circularity: the ontology-internal RAG evaluation matches the paper's stated objective of retrieving ontology-specific definitions.
full rationale
The paper's only potentially self-referential step is the RAG evaluation in Section 5.1: the Ragas ground-truth file is built from competency questions whose answers are extracted from the reified GRhOOT ontology, and the retrieval corpus is the same reified GRhOOT ontology. This makes the evaluation internal, in that context recall and faithfulness measure recovery of ontology content rather than external linguistic correctness. However, this is not circularity in the prohibited sense: the paper explicitly states that the LLM should respond with the ontology's own definitions to ensure consistent annotation guidelines, so ontology-derived ground truth is the appropriate target for the stated objective. The competency questions are not defined in terms of RAG outputs, no fitted parameter is relabeled as a prediction, and the choice among chunking and reranking settings is an honest model-selection comparison rather than a derivation. The reification in Section 3 is a presentational restructuring of the authors' prior GRhOOT ontology, not a derived result claiming independent empirical content. The Section 7 limitation about metaphors and the absence of a user study are validity/scope concerns, not circular steps. Self-citations to prior ontology work are antecedent contributions, not load-bearing uniqueness claims. No enumerated circularity pattern is present.
Assumptions & free parameters
free parameters (4)
- chunk_size =
2048 (basic chunking)
- retrieval_top_k and rerank_k =
12 then 6 (setting alpha)
- LLM temperature =
0.1
- embedding and reranker models =
bge-m3 / bge-reranker-large
assumptions (4)
- domain assumption The GRhOOT ontology's definitions and relations for 110 German rhetorical figures are accurate enough to guide annotation.
- ad hoc to paper The reified properties hasOperation, affectedElement, and hasOperationForm are sufficient and intuitive for users without linguistic training to describe rhetorical figures.
- domain assumption Competency questions generated by gpt-4 from a single figure formalization are a valid evaluation set.
- domain assumption Ragas LLM-based metrics faithfully measure answer quality.
Cite this review
Pith. "Pith review of Enhancing Rhetorical Figure Annotation: An Ontology-Based Web Application with RAG Integration." pith.science (2026). https://pith.science/paper/Z625TFAD
@misc{pith2026241213799,
author = {Pith},
title = {Pith review of: Enhancing Rhetorical Figure Annotation: An Ontology-Based Web Application with RAG Integration},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z625TFAD}},
note = {Machine review of arXiv:2412.13799}
}
read the original abstract
Rhetorical figures play an important role in our communication. They are used to convey subtle, implicit meaning, or to emphasize statements. We notice them in hate speech, fake news, and propaganda. By improving the systems for computational detection of rhetorical figures, we can also improve tasks such as hate speech and fake news detection, sentiment analysis, opinion mining, or argument mining. Unfortunately, there is a lack of annotated data, as well as qualified annotators that would help us build large corpora to train machine learning models for the detection of rhetorical figures. The situation is particularly difficult in languages other than English, and for rhetorical figures other than metaphor, sarcasm, and irony. To overcome this issue, we develop a web application called "Find your Figure" that facilitates the identification and annotation of German rhetorical figures. The application is based on the German Rhetorical ontology GRhOOT which we have specially adapted for this purpose. In addition, we improve the user experience with Retrieval Augmented Generation (RAG). In this paper, we present the restructuring of the ontology, the development of the web application, and the built-in RAG pipeline. We also identify the optimal RAG settings for our application. Our approach is one of the first to practically use rhetorical ontologies in combination with RAG and shows promising results.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Tosin P Adewumi, Roshanak Vadoodi, Aparajita Tripathy, Konstantina Nikolaidou, Foteini Liwicki, and Marcus Liwicki. 2021. Potential idiomatic expression ( PIE )-english: Corpus for classes of idioms. arXiv preprint arXiv:2105.03280
work page Pith review arXiv 2021
-
[2]
Reham Alharbi, Valentina Tamma, Floriana Grasso, and Terry Payne. 2023. An experiment in retrofitting competency questions for existing ontologies. arXiv preprint arXiv:2311.05662
work page Pith review arXiv 2023
-
[3]
Dean Allemang and James Hendler. 2011. Semantic web for the working ontologist: effective modeling in RDFS and OWL . Elsevier
work page 2011
-
[4]
Gloria Italiano Anzilotti. 1982. The rhetorical question as an indirect speech device in English and Italian . Canadian Modern Language Review, 38(2):290--302. Publisher: University of Toronto Press
work page 1982
-
[5]
Shohini Bhattasali, Jeremy Cytryn, Elana Feldman, and Joonsuk Park. 2015. Automatic identification of rhetorical questions. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: Short papers) , pages 743--749
work page 2015
-
[6]
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. Flute: Figurative language understanding through textual explanations. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 7139--7159
work page 2022
-
[7]
Simon Chesterman. 2024. Good models borrow, great models steal: intellectual property rights and generative ai. Policy and Society, page puae006
work page 2024
-
[8]
Fiorela Ciroku, Jacopo de Berardinis, Jongmo Kim, Albert Merono Penuela, Valentina Presutti, and Elena Simperl. 2024. Revont: Reverse engineering of competency questions from knowledge graphs via language models. Journal of Web Semantics
work page 2024
Show all 47 references
-
[9]
Marie Dubremetz and Joakim Nivre. 2015. Rhetorical figure detection: The case of chiasmus. In Proceedings of the fourth workshop on computational linguistics for literature, pages 23--31
2015
-
[10]
Marie Dubremetz and Joakim Nivre. 2017. https://aclanthology.org/W17-0205 Machine learning for rhetorical figure detection: More chiasmus with less annotation . In Proceedings of the 21st nordic conference on computational linguistics, pages 37--45, Gothenburg, Sweden. Associa...
2017
-
[11]
Sanjeev M Dwivedi and Sunil B Wankhade. 2021. Survey on fake news detection techniques. In Image processing and capsule networks: ICIPCN 2020 , pages 342--348. Springer
2021
-
[12]
Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217
2023 arXiv
-
[13]
Yong Fang, Jian Gao, Cheng Huang, Hua Peng, and Runpu Wu. 2019. Self multi-head attention-based convolutional neural networks for fake news detection. PloS one, 14(9):e0222713. Publisher: Public Library of Science San Francisco, CA USA
2019
-
[14]
Simona Frenda, Viviana Patti, and Paolo Rosso. 2023. When sarcasm hurts: Irony -aware models for abusive language detection. In International conference of the cross-language evaluation forum for european languages, pages 34--47. Springer
2023
-
[15]
Robert H Gass and John S Seiter. 2022. Persuasion: Social influence and compliance gaining . Routledge
2022
-
[16]
Martha Gavidia, Patrick Lee, Anna Feldman, and Jing Peng. 2022. Cats are fuzzy pets: A corpus and analysis of potentially euphemistic terms. arXiv preprint arXiv:2205.02728
2022 arXiv
-
[17]
Michael Grüninger and Mark S Fox. 1995. Methodology for the design and evaluation of ontologies. Publisher: Citeseer
1995
-
[18]
Randy Allen Harris and Chrysanne Di Marco. 2017. Rhetorical figures, arguments, computation. Argument & Computation, 8(3):211--231. Publisher: IOS Press
2017
-
[19]
Randy Allen Harris, Chrysanne Di Marco, Sebastian Ruan, and Cliff O’Reilly. 2018. An annotation scheme for rhetorical figures. Argument & Computation, 9(2):155--175
2018
-
[20]
Maia Hristozova and Leon Sterling. 2002. An eXtreme method for developing lightweight ontologies . In Workshop on Ontologies in Agent Systems, 1st International Joint Conference on Autonomous Agents and Multi-Agent Systems,(Bologna, Italy, 2002)
2002
-
[21]
Ashley R Kelly, Nike A Abbott, Randy Allen Harris, Chrysanne DiMarco, and David R Cheriton. 2010. Toward an ontology of rhetorical figures. In Proceedings of the 28th ACM international conference on design of communication , pages 123--130
2010
-
[22]
Ramona K \"u hn and Jelena Mitrovi \'c . 2024 a . The elephant in the room: Ten challenges of computational detection of rhetorical figures. In Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), pages 45--52
2024
-
[23]
Ramona K \"u hn and Jelena Mitrovi \'c . 2024 b . https://doi.org/10.5281/zenodo.10698380 Status Quo der Entwicklungen von Ontologien Rhetorischer Figuren in Englisch , Deutsch und Serbisch . In Book of Abstracts - DHd2024 . Zenodo
2024 doi
-
[24]
Ramona K \"u hn, Jelena Mitrovi \'c , and Michael Granitzer. 2023. Esther: Ontology of rhetorical figures in english. In Proceedings of the Joint Ontology Workshops 2023 Episode IX: The Quebec Summer of Ontology co-located with the 13th International Conference on Formal Ontol...
2023
-
[25]
Ramona K \"u hn, Jelena Mitrovi \'c , and Michael Granitzer. 2024 a . Computational approaches to the detection of lesser-known rhetorical figures: A systematic survey and research challenges. arXiv preprint arXiv:2406.16674
2024 arXiv
-
[26]
Ramona K \"u hn, Khouloud Saadi, Jelena Mitrovi \'c , and Michael Granitzer. 2024 b . Using pre-trained language models in an end-to-end pipeline for antithesis detection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resource...
2024
-
[27]
Ramona Kühn, Jelena Mitrovic, and Michael Granitzer. 2023. Hidden in plain sight: Can german wiktionary and wordnets facilitate the detection of antithesis? In Proceedings of the 12th global wordnet conference, pages 106--116
2023
-
[28]
Ramona Kühn, Jelena Mitrović, and Michael Granitzer. 2022. https://aclanthology.org/2022.lrec-1.426 GRhOOT : Ontology of rhetorical figures in German . In Proceedings of the thirteenth language resources and evaluation conference, pages 4001--4010, Marseille, France. European ...
2022
-
[29]
Jens Lemmens, Ilia Markov, and Walter Daelemans. 2021. Improving hate speech type and target detection with hateful metaphor features. In Proceedings of the fourth workshop on NLP for internet freedom: Censorship , disinformation, and propaganda , pages 7--16
2021
-
[30]
u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...
2020
-
[31]
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173
2024
-
[32]
Miljana Mladenovi \'c , Cvetana Krstev, Jelena Mitrovi \'c , and Ranka Stankovi \'c . 2017. Using lexical resources for irony and sarcasm classification. In Proceedings of the 8th Balkan Conference in Informatics, pages 1--8
2017
-
[33]
Miljana Mladenovi \'c , Jelena Mitrovi \'c , and Cvetana Krstev. 2014. Developing and maintaining a wordnet: Procedures and tools. In Proceedings of the Seventh Global Wordnet Conference, pages 55--62
2014
-
[34]
Miljana Mladenović and Jelena Mitrović. 2013. Ontology of rhetorical figures for serbian. In Text, speech, and dialogue, pages 386--393, Berlin, Heidelberg. Springer
2013
-
[35]
Natalya F Noy, Deborah L McGuinness, and others . 2001. Ontology development 101: A guide to creating your first ontology
2001
-
[36]
Cliff O’Reilly, Yetian Wang, Katherine Tu, Sarah Bott, Paulo Pacheco, Tyler William Black, and Randy Allen Harris. 2018. Arguments in gradatio, incrementum and climax; a climax ontology. In Proceedings of the18th workshop on computational models of natural argument. Academic press
2018
-
[37]
Suhas Ranganath, Xia Hu, Jiliang Tang, Suhang Wang, and Huan Liu. 2018. Understanding and identifying rhetorical questions in social media. ACM Transactions on Intelligent Systems and Technology (TIST), 9(2):1--22
2018
-
[38]
Victoria L Rubin, Niall Conroy, Yimin Chen, and Sarah Cornwell. 2016. Fake news or truth? using satirical cues to detect potentially misleading news. In Proceedings of the second workshop on computational approaches to deception detection, pages 7--17
2016
-
[39]
Jan Smits and Tijn Borghuis. 2022. Generative ai and intellectual property rights. In Law and artificial intelligence: regulating AI and applying AI in legal practice, pages 323--344. Springer
2022
-
[40]
Robert Stevens and Phillip Lord. 2010. https://ontogenesis.knowledgeblog.org/993/ Reification of properties in an ontology . Ontogenesis, An Ontology Tutorial
2010
-
[41]
Claus Walter Strommer. 2011. Using rhetorical figures and shallow attributes as a metric of intent in text. Publisher: University of Waterloo
2011
-
[42]
Enrica Troiano, Carlo Strapparava, Gözde Özbal, and Serra Sinem Tekiroğlu. 2018. A computational exploration of exaggeration. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3296--3304
2018
-
[43]
Yetian Wang, Randy Allen Harris, and Daniel M Berry. 2021. An ontology for ploke: Rhetorical figures of lexical repetitions. In JOWO
2021
-
[44]
Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. 2024. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473
2024 arXiv
-
[45]
Dawei Zhu, Qiusi Zhan, Zhejian Zhou, Yifan Song, Jiebin Zhang, and Sujian Li. 2022. ConFiguRe : Exploring discourse-level chinese figures of speech. arXiv preprint arXiv:2209.07678
2022 arXiv
-
[46]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[47]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.