{"id":"020f84b3-3be4-4dea-b72f-c84ee630ba5d","arxiv_id":"2411.18260","paper_version":3,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors present MetaphorShare, a public repository that unifies 25 English metaphor datasets under a common annotation format with upload, search, and labeling tools.","lead":"MetaphorShare is a new website that lets researchers upload, download, search, and label metaphor datasets in a common format. It aims to connect the metaphor studies and NLP communities by making labeled metaphor corpora easier to find and reuse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Format preservation is asserted but not demonstrated; it is the weakest point of an otherwise sound system paper.","rationale":"The paper's central claim is a live, usable repository with four main functionalities. Screenshots, a system description, and a self-reported sanity check of upload, database insertion, and search support the platform claim, so the main contribution is credible. The weakest technical assumption is the lossless conversion to the unified CSV format; the reader identified this same assumption, and I agree it is load-bearing because the repository's research value depends on datasets remaining usable. However, the paper's flexible design (free columns, the free <u> tag, and two MIPVU variants) makes full information preservation plausible, and no concrete lossy mapping is demonstrated. The missing round-trip verification is an important limitation but not a fatal flaw. Therefore the ACCEPT verdict stands unchanged; a simple round-trip audit would materially strengthen the preservation claim.","tokens_in":12725,"tokens_out":11540,"duration_ms":109096,"concrete_test":"Run a round-trip fidelity audit: take an original MIPVU-annotated document (e.g., from VUAC_BO), apply the same conversion pipeline used to upload to MetaphorShare, download the resulting CSV via the site's search/download interface, and attempt to reconstruct the original annotation for every token, including non-binary MIPVU categories and any continuous scores. Repeat for a dataset with continuous variables such as Mohammad et al. (2016). If any original label or score is unrecoverable, the Section 1 preservation claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing point is the assertion in Section 1 that unifying datasets preserves the information encoded in the original resources. The five-tag CSV format (<m>, <l>, <t>, <a>, <u>) and optional free columns are flexible, but no exact mapping from original annotation schemes is specified. This matters for MIPVU-derived datasets: MIPVU labels every token and distinguishes categories beyond a binary metaphor/literal split, while the unified tag set has only <m>/<l> plus a free <u>. Whether the original categories and any continuous variables (metaphoricity, emotion ratings) survive conversion is undocumented. The two MIPVU versions mentioned in Section 3 are two possible representations, not evidence of losslessness. If conversion silently drops distinctions, users of the repository cannot trust the data, and the cross-dataset experiments in Section 5 inherit that risk. This concern is not a demonstrated failure, and the flexible format could in principle preserve everything, but the paper does not provide the round-trip evidence needed to establish the preservation claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MetaphorShare, a web-based repository for metaphor datasets, and describes its four main functionalities: upload, download, search, and label. The authors argue that the platform unifies heterogeneous metaphor annotation formats into a single CSV schema with tags <m>, <l>, <t>, <a>, and <u>, while preserving the information in the original datasets. The paper reports that 25 English datasets are currently integrated, describes the system architecture (FastAPI backend, Elasticsearch, PostgreSQL, React frontend), and presents a case study in which RoBERTa models are fine-tuned on twelve of the datasets for cross-dataset metaphor identification. The central claim is that MetaphorShare is a functioning, open, collaborative resource that makes metaphor datasets more accessible and interoperable.","tokens_in":12788,"tokens_out":3490,"duration_ms":32840,"significance":"If the platform works as described, it addresses a genuine gap: many metaphor-labelled resources are scattered, inconsistently formatted, or not known outside small communities. The paper provides a concrete system description, public access, and an illustration of how the repository can support comparative NLP experiments. The strengths are the clear architecture description, the inclusion of diverse datasets, and the fact that the platform is openly accessible with a unified searchable format. However, the paper's own evidence for the preservation of original annotation information is incomplete, and the cross-dataset evaluation is too under-specified to support its comparative claims. These issues are fixable and do not undermine the basic utility of the resource, but they need to be addressed before the paper is ready for publication.","major_comments":[{"comment":"The paper states in Section 1 that the repository preserves 'the information encoded in the original datasets,' but Section 3 does not provide a mapping from the original annotation schemes to the five-tag CSV format. In particular, MIPVU-derived resources label every token and use categories beyond a binary metaphor/literal distinction, and several datasets include continuous variables such as metaphoricity, novelty, or emotion ratings plus document-level context. The free <u> tag and open columns are flexible, but no example or round-trip check demonstrates that these survive conversion; the two VUAC versions (multiple tags per sentence vs. one tag with duplicated sentences) suggest that choices are being made that could lose information. I request an explicit conversion protocol per dataset type, or a statement of which information is normalised away, and ideally a reconstruction test showing that the original files can be recovered from the stored records.","section":"Section 3, 'Unified input format'"},{"comment":"Section 3 says the twelve illustrative datasets have open licenses, but Table 4 lists PVC as having 'no license.' This contradicts the paper's claim of an open repository and raises legal questions about redistribution. The authors should clarify whether PVC is actually included in MetaphorShare and under what terms, or replace it with a properly licensed dataset in the list of representative examples.","section":"Section 3 and Table 4"},{"comment":"The cross-dataset evaluation is presented as a case study, but it reports a single F1 score per condition without stating random seeds, number of runs, error bars, or significance tests. As a result, statements such as 'A few datasets generalise better than others' and 'TONG and NEWS do not generalize as well' are not supported by the evidence shown. I suggest either adding repeated runs with confidence intervals, or explicitly describing the figure as a single illustrative run and softening the comparative claims.","section":"Section 5, 'Experimental setting' and 'Results'"}],"minor_comments":[{"comment":"The text 'decide weather the marked expression' should be 'decide whether the marked expression is used metaphorically or literally.'","section":"Section 3"},{"comment":"The sentence listing multilingual examples contains two unresolved placeholders: 'Mandarin Chinese sentences in ?' and 'Farsi sentences in ?'. Please cite the relevant datasets or remove these examples.","section":"Section 3, 'Unified input format'"},{"comment":"The JANK entry contains a duplicated word: 'because because they are not used.'","section":"Appendix A"},{"comment":"The license for TSV_A is given as 'see data page' without a URL in the table; please include the actual license name or a stable link.","section":"Table 4"},{"comment":"The phrase 'restricted search for a label' is ambiguous; consider rephrasing to 'the label can be filtered to metaphorical, literal, anomalous, or other categories.'","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system/resource demonstration, and the core contribution — a working, publicly accessible repository — is valuable and within the scope of the venue. The reader's acceptance is understandable, but I believe the format-preservation claim and the license inconsistency need to be resolved before acceptance. The evaluation section, while illustrative, should not be presented with unsupported comparative statements. These are fixable within the manuscript's scope, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nMetaphorShare is a real contribution: a central, searchable repository for open metaphor datasets with upload, download, search, and a simple annotation tool, all unified under a five-tag CSV format. That is genuinely new relative to MetaPro and Metaphor Magnet, which tag free text but don't aggregate existing labeled resources. The paper also does the right thing by scoping the cross-dataset experiments as a usage illustration, not a rigorous benchmark.\n\nThe main soft spot, and the one the stress test rightly flags, is the preservation claim. Section 1 says the repository unifies formats 'while preserving the information encoded in the original datasets,' but the paper never demonstrates that the five-tag format (<m>, <l>, <t>, <a>, <u>) plus free columns can represent every original annotation scheme. The discussion of MIPVU is the clearest case: MIPVU labels every token and distinguishes categories beyond binary metaphor/literal, and the paper says datasets are integrated in two versions but doesn't show the mapping is lossless. The sanity check tests that upload, database insertion, and search work; it does not test that round-tripping a dataset through the format preserves its original distinctions. This is not a demonstrated failure, and the flexible format could in principle preserve everything, but the paper needs to document the mapping or at least state which original fields are preserved and which are collapsed. Without that, downstream users can't trust the repository for anything beyond the binary classification tasks.\n\nA few minor issues: the manuscript has placeholder citations ('Mandarin Chinese sentences in ?, Farsi sentences in ?') in Section 3, a duplicated 'because because' in the appendix, and the cross-dataset F1 scores come without error bars or seeds. None of these are load-bearing, but they should be cleaned up.\n\nOverall, this is a solid, honest system paper. The resource itself is useful for metaphor processing researchers, and the paper's central claim—that the website works—is supported by the provided evidence. It deserves a serious referee. I'd recommend acceptance after a minor revision that addresses the format-preservation documentation and the placeholders.","headline":"A useful, honestly scoped repository paper; the main soft spot is the unproven claim that the unified format preserves all original dataset information.","tokens_in":13394,"tokens_out":2281,"would_cite":false,"duration_ms":20861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new web repository, MetaphorShare, unifies 25 open metaphor datasets under one searchable and re-usable format, with upload, download, search, and labeling tools built in.","keywords":["metaphor datasets","open repository","metaphor identification","dataset format unification","MIPVU","annotation tool","cross-dataset evaluation","Elasticsearch"],"falsifier":"Download a converted dataset from the repository and compare it against the authors' original release; any instance whose document-level context, multiple labels per span, or continuous score (e.g., metaphoricity rating) is missing or altered would falsify the claim that the unified format preserves original information.","tokens_in":12435,"feed_emoji":"🗂️","tokens_out":4088,"duration_ms":33625,"temperature":0.7,"pith_summary":"MetaphorShare is a live website that collects metaphor-labelled corpora from different research traditions and makes them available through a common interface. The paper argues that a single, minimally constrained CSV format can hold datasets that differ in annotation units, context length, and additional variables, and that the repository preserves the original information. If this works, researchers in linguistics and NLP no longer have to hunt for scattered or unpublished datasets or reformat them by hand, and metaphor-identification systems can be trained and evaluated across many datasets at once. The platform currently hosts 25 English datasets and includes an online annotation tool whose output feeds directly into the repository.","feed_headline":"MetaphorShare unifies 25 open datasets in one searchable hub","feed_subtitle":"Upload, download, search, and label metaphor data under a single format to speed up NLP and linguistics research.","key_machinery":"The load-bearing device is the unified CSV format: each line must contain a tagged_text column with at least one tagged expression, using the tag set <m>, <l>, <t>, <a>, <u>, and optional extra columns for sentence index, reference, part of speech, target, source/target domain, metaphoricity scores, or free fields. This format is what lets the repository compare, search, and download heterogeneous datasets through one database schema. Around it sits the website architecture: a FastAPI backend, a PostgreSQL store for dataset metadata, an Elasticsearch index of 'potentially metaphoric expressions' that powers exact and fuzzy multilingual search, and a browser annotation tool that emits the same CSV format.","core_discovery":"The paper presents MetaphorShare as a functioning open repository, currently holding 25 English metaphor datasets, organised around four functionalities: upload, download, search, and label. Its central claim is that a minimally constrained CSV format with five predefined tags — <m> for metaphoric, <l> for literal, <t> for target cue, <a> for anomalous, and <u> for free user-defined tagging — plus optional named columns for continuous or categorical variables, is flexible enough to integrate datasets spanning psycholinguistic word norms, NLP binary classification, multi-word idiom detection, and MIPVU-based full-corpus annotation, without destroying the information the original datasets encode. The paper supports this by converting twelve representative datasets, describing the Elasticsearch-based search and PostgreSQL-backed upload pipeline, and running a cross-dataset RoBERTa metaphor-identification experiment to show that the repository makes multi-dataset experimentation straightforward.","pith_inferences":["If the repository grows to include non-English datasets as planned, the same CSV format and Elasticsearch multilingual search could make cross-lingual metaphor studies directly comparable, something the current English-only collection only gestures at.","The five-tag system may end up functioning as a de facto interchange standard for metaphor annotation, analogous to what CoNLL formats did for syntactic annotation; that would be a larger consequence than the paper itself claims.","A natural stress test would be to upload datasets with overlapping but distinct annotation guidelines (MIPVU vs. idiom detection) and measure whether the unified search index supports reliable label-consistent retrieval; the paper does not run that test."],"forward_implications":["New datasets formatted as tagged CSV can be uploaded by any user, and after automatic format validation plus manual license review they appear in the catalog and search index.","A cross-dataset evaluation on ten training sets shows that models fine-tuned on one metaphor dataset transfer to others to varying degrees, with short, syntactically constrained sets (J&C, GUT) generalising unexpectedly well.","The search page allows filtering by dataset, language, and label, with exact and fuzzy matching on tagged expressions or full text, and any result set can be downloaded as CSV.","The annotation tool produces labels directly compatible with the repository format, lowering the barrier for annotators without programming experience."],"supporting_citations":[{"why":"Defines the MIPVU procedure and the VU Amsterdam Corpus, the model for multi-tag, discourse-level metaphor annotation that the repository format must accommodate.","marker":"Steen (2010)"},{"why":"Provides the VUAC_BO dataset and the BOHB-based experimental setup reused in the Section 5 cross-dataset evaluation.","marker":"Boisson et al. (2023)"},{"why":"Supplies RoBERTa base, the model fine-tuned independently on each training set in the evaluation.","marker":"Liu et al. (2019)"},{"why":"Supplies the BOHB hyperparameter optimization algorithm used to tune the models in the evaluation.","marker":"Falkner et al. (2018)"},{"why":"An integrated dataset with continuous emotion scores, used as the illustrative search-page record for optional additional fields.","marker":"Mohammad et al. (2016)"},{"why":"MAGPIE, the largest integrated dataset, exemplifies the long-context, multi-word expression annotation style the format must handle.","marker":"Haagsma et al. (2020)"}],"fun_headline_variants":["MetaphorShare: 25 datasets, one format, four tools","Centralize metaphor data: upload, search, label, download","All metaphor datasets, one repo: share and search","MetaphorShare: the collaborative repo for metaphor NLP","Break metaphor data silos with MetaphorShare's open format"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The unified CSV format with its five tags and optional columns preserves all the information contained in the original metaphor datasets, including multiple annotations per expression, document-level context, and continuous variables such as metaphoricity scores.","fun_headline_variants_meta":{"raw":{"variants":["MetaphorShare: 25 datasets, one format, four tools","Centralize metaphor data: upload, search, label, download","All metaphor datasets, one repo: share and search","MetaphorShare: the collaborative repo for metaphor NLP","Break metaphor data silos with MetaphorShare's open format"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001144,"raw_usage":{"total_tokens":4703,"prompt_tokens":857,"completion_tokens":3846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":3761}},"tokens_in":473,"tokens_out":3846,"duration_ms":27420,"temperature":1.0,"reasoning_tokens":3761,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:21:20.864617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Download a converted dataset from the repository and compare it against the authors' original release; any instance whose document-level context, multiple labels per span, or continuous score (e.g., metaphoricity rating) is missing or altered would falsify the claim that the unified format preserves original information.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MIPVU procedure and the VU Amsterdam Corpus, the model for multi-tag, discourse-level metaphor annotation that the repository format must accommodate."}],"review_version":1}