{"id":"bbfc8b63-1c9e-4b06-9857-45f988d95109","arxiv_id":"2505.08690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Paraphrasing event schemas and retrieving the most relevant ones improves LLM-based event extraction, as shown on the new 12-dataset MD-SEE benchmark.","lead":"This paper proposes a retrieval-augmented method that paraphrases event-extraction schemas before retrieving them, plus a new benchmark that combines 12 existing event-extraction datasets. The method improves schema retrieval and extraction accuracy across domains, though the end-to-end evaluation has some weaknesses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 5 ignores false-positive retrieved schemas, so retrieval precision does not affect reported end-to-end scores; the claim that ASEE significantly improves extraction accuracy rests on a metric that cannot detect the main failure mode it claims to address.","rationale":"The reader identified Equation 5 as the weakest assumption, and this stress-test agrees. The retrieval-side evidence (Tables 1 and 2) is consistent and supports the narrower claim that paraphrasing improves schema recall; that part of the paper is not the target of this objection. The load-bearing issue is specifically the end-to-end claim. Equation 5's denominator is fixed at the number of ground-truth schemas, so false-positive retrievals are costless by construction. This is not merely a philosophical preference for precision: it means the metric cannot distinguish a system that finds the right schema from one that floods the prompt with every candidate. Deployment harms such as spurious event types are exactly the outcomes the paper says it prevents, yet the metric makes them invisible. The proposed recomputation is a single, feasible check that would show whether the headline improvement survives a precision-sensitive evaluation. If it survives, the conditional acceptance can stand; if it does not, the paper still contributes the benchmark and retrieval findings, but the end-to-end claim needs to be softened or re-evaluated.","tokens_in":16651,"tokens_out":6410,"duration_ms":62599,"concrete_test":"Recompute all rows of Table 4 (and the per-dataset end-to-end tables in Appendix B) with a precision-aware variant E2E-F1* = (1/N) sum_q (2 * sum_{s in S_q} F1(s,q) * I(s in R_q)) / (|S_q| + |R_q|), which penalizes every retrieved schema that is not ground truth, and also report precision@k. If the ordering of retrieval models or the paraphrase-vs-raw gap changes materially, Equation 5 was the source of the claimed improvement. As a degenerate-case check, run the same extractor with k=300: if the Equation 5 score approximates oracle retrieval, the metric is confirmed to be precision-blind.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The end-to-end evaluation in Section 5.5 uses the modified E2E-F1 in Equation 5, which averages extraction F1 over ground-truth schemas S_q and multiplies by I(s in R_q). Retrieved schemas outside S_q never enter the numerator or denominator. This makes the score invariant to retrieval precision: a retriever that returns the entire 300-schema pool receives the same credit for every true schema as a retriever with perfect precision, provided the extractor's F1 on true schemas is unaffected by the extra prompt content. In the limit, E2E-F1 measures recall-weighted extraction quality, not joint retrieval-and-extraction accuracy. Because the central claim that schema paraphrasing improves end-to-end event extraction depends on Table 4 and Appendix B, and because false-positive schemas in a real deployment would trigger spurious extractions that this metric ignores, the stated end-to-end gains are not established by the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASEE, an adaptive schema-aware event extraction framework that combines schema paraphrasing with retrieval-augmented generation. Given a query, ASEE first paraphrases raw event schemas into a richer schema pool using few-shot training examples, then retrieves the top-k relevant schemas from a pool of hundreds, and finally extracts event arguments using an LLM guided by those retrieved schemas. To evaluate the joint task of schema selection and extraction, the authors construct MD-SEE, a benchmark that consolidates 12 existing datasets across domains, languages, and query lengths. Experiments report improved schema retrieval Recall@K with paraphrased schemas and an end-to-end E2E-F1 score on MD-SEE and individual datasets, claiming significant accuracy improvements over raw-schema pipelines. The paper also provides schema-aware extraction comparisons for several open-source LLMs and GPT-4-turbo.","tokens_in":16854,"tokens_out":4508,"duration_ms":41724,"significance":"If the claims hold, ASEE would be a useful practical contribution: it addresses the real-world need to select schemas from a large candidate pool before extraction, and MD-SEE provides a benchmark for the joint schema-matching and extraction setting. The paper also ships extensive empirical comparisons and a reproducible benchmark construction pipeline. The main scientific interest is the schema-paraphrasing idea, which appears to substantially help retrieval in Tables 1 and 2. However, the end-to-end evaluation is weakened by the choice of metric, which ignores retrieval precision, and by the absence of a raw-schema end-to-end baseline on MD-SEE. These issues affect the central claim that ASEE 'significantly improves' event extraction in the joint setting, so the current evidence is not yet convincing.","major_comments":[{"comment":"The E2E-F1 metric ignores all retrieved schemas that are not in the ground-truth set S_q. Consequently, a retriever that returns the entire 300-schema pool receives the same credit for each ground-truth schema as a perfectly precise retriever, and retrieval precision does not affect the reported score. Under this metric E2E-F1 is a recall-weighted extraction quality measure, not a joint retrieval-and-extraction accuracy measure. Because the paper's central end-to-end claim (Section 5.5, Abstract) is that schema paraphrasing improves the combined retrieval-and-extraction pipeline, the metric must penalize false-positive retrievals. Please re-evaluate with a metric that counts false-positive retrievals (for example, by scoring extractions made under unrequested schemas as false positives, or by using a joint F1 over all retrieved schemas).","section":"Section 5.1.3, Eq. (5)"},{"comment":"The MD-SEE end-to-end evaluation compares different retrievers and extraction models, but every row uses the paraphrased schema pool; there is no raw-schema end-to-end baseline on MD-SEE. Thus Table 4 cannot by itself support the claim that schema paraphrasing improves end-to-end extraction. The per-dataset tables in Appendix B do provide Raw versus Paraph. comparisons, but those results are not uniformly favorable to paraphrasing: for example, in Table 7 (CrudeOilNews) several retrievers perform equal or slightly worse with Paraph. (BM25 0.07 vs 0.08; BGE-M3 0.07 vs 0.08; E5-LV2 equal at 0.08). Please provide the raw-schema end-to-end comparison on MD-SEE and discuss the per-dataset variability, which the current narrative glosses over.","section":"Section 5.5, Table 4 "},{"comment":"The text states that MD-SEE contains 12,817 training samples, 1,775 development samples, and 7,686 test samples, but Table 6 reports a development size of 800 in the total row, and the per-source development counts sum to 800. This inconsistency makes the exact split of the benchmark ambiguous. Please correct the numbers and ensure the text, tables, and any released data agree.","section":"Section 4.2.3 vs Table 6"}],"minor_comments":[{"comment":"The phrase 'we developed theMulti-DimensionalSchema-awareEventExtraction (MD-SEE) benchmark' contains a doubled 'we developed the'; please rewrite for clarity.","section":"Section 1, paragraph 2"},{"comment":"The description of the schema-aware extraction F1 metric does not specify the matching granularity (e.g., exact string, token-level, or span-level). Specifying this is important for reproducibility, especially for multilingual and document-level queries.","section":"Section 5.1.2"},{"comment":"The schema consolidation uses an ad-hoc cosine similarity threshold of 0.85 and a greedy independent-set heuristic; no sensitivity analysis is reported. Since the composition of the schema pool directly affects all subsequent retrieval and extraction results, an ablation on this threshold would strengthen the paper.","section":"Section 4.2.1 and Appendix A.3"},{"comment":"The phrasing 'significantly improving the accuracy of event extraction' in the Abstract and Section 5.5 is not supported by any statistical significance tests. The recorded differences (e.g., BGE-M3 E2E-F1 0.62 with Llama-3.2-3B vs 0.46 for BM25) may be meaningful, but at least a paired significance test should be reported given the large test set.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The central methodological idea is sound and the retrieval evaluations in Tables 1 and 2 are convincing on their own. The main barrier to acceptance is the end-to-end evaluation: the E2E-F1 metric as defined in Eq. (5) gives no credit or penalty for retrieving irrelevant schemas, which makes the metric insensitive to the exact failure mode the paper claims to solve. Additionally, the missing raw-schema end-to-end baseline on MD-SEE and the inconsistent dev-set counts are fixable but currently undermine the paper's main claim. I do not see a circularity problem: paraphrased schemas are generated from training data only, and the retrieval/extraction models are not fitted to test labels. The authors should be encouraged to revise with a stricter joint metric and a raw-schema end-to-end comparison; the paper would then be a solid contribution to the event extraction and RAG literature. I would not require the authors to fully resolve the metric debate; a thorough discussion of the metric's behavior plus a supplementary precision-based evaluation would suffice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: schema paraphrasing is a genuinely simple, useful idea, and the retrieval numbers back it up. But the end-to-end claim in the abstract and Section 5.5 does not survive contact with Eq. 5.\n\nWhat is new: rewriting each raw schema into a richer, more retrievable form using a few training examples, then retrieving schemas and extracting with an LLM. The Recall@K gains in Tables 1 and 2 are consistent across seven retrievers and multiple languages; that is a real contribution. MD-SEE is also a sensible consolidation of 12 datasets into 300 schemas, though the paper gives no release link, which limits its immediate reuse.\n\nThe soft spots are in the evaluation. E2E-F1 (Eq. 5) averages extraction F1 over ground-truth schemas and multiplies by an indicator that the schema was retrieved. Schemas retrieved but not in the ground truth never enter the score. So a retriever that returns the whole 300-schema pool gets full credit for every true schema, as long as extraction F1 on those true schemas is unchanged by the extra prompt content. In other words, the metric is recall-weighted extraction quality, not joint retrieval-and-extraction accuracy. The stress-test note is correct: the headline end-to-end improvement is not established by these numbers. Real deployments would pay for false-positive schema retrievals via spurious extractions, and this metric is blind to that cost.\n\nThere are two further problems. Table 4 (the MD-SEE end-to-end result) has no raw-schema baseline, so even under the lenient metric, you cannot attribute the end-to-end score to paraphrasing rather than to the retriever. The per-dataset appendix tables do have Raw vs Paraph., but the results are mixed—paraphrasing hurts on some datasets (e.g., CrudeOilNews with several retrievers). And the dev-set statistics are inconsistent: Section 4.2.3 reports 1,775 dev samples, while Table 6 sums to 800. Sloppy, and should be fixed.\n\nThe retrieval-side evidence is genuine, and the framework is not circular: paraphrases are generated from training data, and SFT uses training labels only. The limitations section is honest about what is not covered. I would send this to peer review, but with a clear expectation of major revision: add a strict end-to-end metric that penalizes false-positive retrievals, include a raw-schema end-to-end baseline on MD-SEE, reconcile the dev counts, and make the benchmark downloadable. The paper is worth refereeing because the core idea and the retrieval result are solid, but the main claim needs to be re-earned.","headline":"Schema paraphrasing clearly helps retrieval, but the reported end-to-end gains rest on a metric that ignores false-positive retrievals and a missing raw-schema baseline.","tokens_in":17394,"tokens_out":4347,"would_cite":false,"duration_ms":38663,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that rewriting event schemas with few-shot examples makes schema retrieval substantially more accurate, and that a retrieval-then-extraction pipeline built on the paraphrased pool improves event extraction across domains.","keywords":["event extraction","schema retrieval","retrieval-augmented generation","schema paraphrasing","large language models","MD-SEE","cross-lingual extraction","event extraction benchmark"],"falsifier":"Compute an end-to-end score that penalizes retrieved-but-irrelevant schemas, for example by including false-positive schemas in the denominator or subtracting a precision term, on MD-SEE with BGE-M3 and Llama-3.1-8B. If the advantage of paraphrased schemas over raw schemas shrinks or reverses under that metric, the practical value of the pipeline depends on the cost of false positives.","tokens_in":16459,"feed_emoji":"🔍","tokens_out":7936,"duration_ms":71739,"temperature":0.7,"pith_summary":"Event extraction in practice requires choosing the right schema from hundreds of candidates before extracting anything, yet current systems either hard-code schemas or stuff all of them into the prompt. This paper proposes Adaptive Schema-aware Event Extraction (ASEE), which paraphrases each schema with few-shot training examples, retrieves the top-k paraphrased schemas for a query, and then extracts events with an LLM fine-tuned to follow those schemas. The paper constructs MD-SEE, a 300-schema, 12-dataset benchmark spanning domains, languages, and query lengths, and reports that paraphrased schemas improve retrieval Recall@10 across all retrievers tested, with BGE-M3 rising from 0.61 to 0.78 on MD-SEE. The paper's claim is that the bottleneck in schema-conditioned event extraction is the terse wording of raw schemas, and that rewriting them into fuller language is a cheap, transferable fix.","feed_headline":"Paraphrased schemas lift event retrieval recall from 0.61 to 0.78","feed_subtitle":"A retrieve-then-extract pipeline handles 300 event schemas across domains without hand-fixing them.","key_machinery":"The load-bearing object is the paraphrased schema pool S. For each raw schema s in S0, the paper uses training samples Ds that instantiate s as few-shot demonstrations and prompts a frozen LLM to rewrite s with fuller argument descriptions; the outputs form S. A retriever ψ, chosen from BM25, BGE-M3, E5, GTE, LLM-Embedder, or reranker variants, scores a query q against S and returns the top-k schemas. A supervised fine-tuned LLM θ then generates argument values from q and the retrieved schemas. The modified E2E-F1 metric given in Eq. 5 is also load-bearing: it averages extraction F1 over ground-truth schemas that were retrieved, assigns zero to gold schemas that were missed, and ignores retrieved schemas that are not in the gold set.","core_discovery":"The central claim is that schema paraphrasing, using a frozen LLM to rewrite each raw schema with a few training examples as demonstrations, creates a schema pool whose entries align with real query language, making retrieval of the correct schema much easier. On MD-SEE, paraphrased schemas beat raw schemas for every one of the seven retrievers at Recall@10, Recall@20, and Recall@50; for instance, BGE-M3 reaches 0.78 Recall@10 on paraphrased schemas versus 0.61 on raw schemas. The paper then shows that a retrieve-then-extract system, where the extractor is an SFT-tuned open-weight LLM and only top-k schemas enter the prompt, achieves higher end-to-end F1 than a pipeline using raw schemas, with larger and fine-tuned extractors compounding the gain. In the paper's framing, the same two-stage recipe carries across news, finance, cybersecurity, biomedical, and legal domains as well as English-Chinese cross-lingual pairs.","pith_inferences":["A testable extension is to measure precision-aware end-to-end accuracy: if every retrieved schema costs compute or confuses downstream users, the reported E2E-F1 may overstate the practical gain, since the metric ignores false-positive retrievals.","The paraphrasing step could serve as a general schema-normalization layer for relation extraction and named entity recognition, not just event extraction, because the same few-shot-rewrite-with-training-samples recipe applies to any structured schema.","A held-out paraphrase experiment would separate surface mimicry from semantic alignment: if retrieval is high only for paraphrases produced from the same training split, the gain may reflect lexical overlap rather than better schema understanding."],"forward_implications":["With paraphrased schemas, even the lexical retriever BM25 becomes usable, jumping from 0.33 to 0.58 Recall@10 on MD-SEE, which lowers the entry cost for retrieval in low-resource deployments.","The pipeline avoids context-window overflow by feeding only top-k schemas to the extractor, so the approach scales to hundreds of candidate schemas without prompt-compression tricks.","Schema consolidation plus a greedy maximum-independent-set step reduces duplicate schemas, so a 300-schema pool still covers 12 datasets without redundant entries.","The ASEE recipe with BGE-M3 as retriever and Llama-3.1-8B with supervised fine-tuning gives the best E2E-F1 on MD-SEE, meaning retrieval quality and extractor capacity both contribute to the gain.","Because the paraphrasing step is data-driven and language-agnostic, ASEE generalizes to Chinese queries with English schemas and English queries with Chinese schemas, suggesting multilingual deployment without extra annotation."],"supporting_citations":[{"why":"Defines schema-aware event extraction (SEE), the task decomposition ASEE builds on and extends with retrieval.","marker":"Shiri et al. (2024)"},{"why":"Provides IEPILE, the schema-conditioned information extraction corpus from which several MD-SEE datasets are drawn.","marker":"Gui et al. (2024)"},{"why":"Supplies BGE-M3, the multilingual embedding model used both to consolidate schemas and as the strongest retrieval model.","marker":"Chen et al. (2024)"},{"why":"Motivates the retrieval-augmented generation design: injecting external schema knowledge into the prompt instead of relying on the LLM's memory.","marker":"Gao et al. (2023)"},{"why":"Supplies BM25, a lexical baseline whose Recall@10 jumps from 0.33 to 0.58 after paraphrasing, demonstrating the central claim even without neural retrieval.","marker":"Robertson and Zaragoza (2009)"},{"why":"Contributes DocEE, the document-level event extraction dataset used for the English and cross-lingual subsets.","marker":"Tong et al. (2022)"},{"why":"Contributes MAVEN-Arg, the argument-extraction dataset with a large set of event types and roles used in MD-SEE.","marker":"Wang et al. (2024a)"},{"why":"Contributes GENEVA, a general-domain event argument extraction dataset used to test schema generalization.","marker":"Parekh et al. (2023)"}],"fun_headline_variants":["Schema paraphrasing lifts event retrieval recall to 0.78","Retrieve-then-extract with paraphrased schemas beats raw pipelines","Frozen LLM rewrites schemas, driving recall from 0.61 to 0.78","Adaptive schema selection via RAG improves event extraction accuracy","Paraphrase schemas first, then retrieve: a winning combo for EE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The end-to-end evaluation assumes that retrieving extra schemas that are not ground truth is harmless, because the score ignores them entirely and only rewards finding the correct ones.","fun_headline_variants_meta":{"raw":{"variants":["Schema paraphrasing lifts event retrieval recall to 0.78","Retrieve-then-extract with paraphrased schemas beats raw pipelines","Frozen LLM rewrites schemas, driving recall from 0.61 to 0.78","Adaptive schema selection via RAG improves event extraction accuracy","Paraphrase schemas first, then retrieve: a winning combo for EE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1567,"prompt_tokens":957,"completion_tokens":610,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":573,"tokens_out":610,"duration_ms":5871,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:47:53.218861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute an end-to-end score that penalizes retrieved-but-irrelevant schemas, for example by including false-positive schemas in the denominator or subtracting a precision term, on MD-SEE with BGE-M3 and Llama-3.1-8B. If the advantage of paraphrased schemas over raw schemas shrinks or reverses under that metric, the practical value of the pipeline depends on the cost of false positives.","supporting_citations":[],"review_version":1}