{"id":"bddf4ab2-f198-460e-a152-3a34d666d514","arxiv_id":"2505.12236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"TKRE combines LLM-generated explanations and synthetic data with two-stage span masking and contrastive pre-training to improve few-shot relation extraction.","lead":"This paper proposes TKRE, a framework that uses large language models to generate explanations and synthetic examples, then pre-trains a small relation extraction model with masked span and contrastive losses. It reports state-of-the-art few-shot relation extraction results on four benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MSLM/SCL span selection is unspecified; the auxiliary objectives may leak the golden relation through LLM-generated explanations, so the claimed contribution of the two-stage pre-training is not yet established.","rationale":"The reader's weakest assumption correctly identifies the missing span-selection specification in Sec. 3.3. My stress-test goes one step further: because the explanation corpus is generated with the golden relation label in the prompt, the missing specification is not just an implementation detail. If the span labels are derived from that explanation, the MSLM and SCL objectives can encode the supervised relation label directly, which would make the two-stage pre-training a form of label leakage rather than knowledge-guided transfer. That would undermine the central claim that the pre-training objectives contribute relational reasoning beyond the synthetic data itself. This is the most load-bearing concern because it targets the mechanism that differentiates TKRE from prior data-augmentation hybrids; the F1 gains could then be an artifact of the extra data or of implicit label supervision. The concern is concrete and testable: the authors release code and data, and an inspection of the span-extraction routine together with a random-span control experiment would settle whether the objectives matter. I do not recommend REJECT because the paper does contain ablation studies and released artifacts; the issue may be fully resolvable. UNCHANGED (still CONDITIONAL) is appropriate, with the condition being that the span-selection procedure and a leakage-free control be provided. The other reviewer concerns, such as missing variance reporting and baseline comparability, are secondary and would not by themselves change the verdict.","tokens_in":17031,"tokens_out":5115,"duration_ms":54714,"concrete_test":"Inspect the released GitHub repository for the code that builds C and assigns relation, entity, and negative spans. Verify whether the LLM prompt includes the golden relation label (it does in Sec. 3.2) and whether span labels are derived from the explanation text via string matching of the relation name or a fixed trigger list. Then run a matched control: replace MSLM/SCL span selection with random spans of identical length and position distribution and random negatives, keeping all other data and losses identical; repeat 8-shot TACRED with 5 seeds. If the control closes the F1 gap to within noise, the two-stage objectives are not the active ingredient; if TKRE still leads by more than 3 F1 points, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the two-stage pre-training objectives in Sec. 3.3. The explanation corpus C is built in Sec. 3.2 by prompting a frozen LLM with the golden relation label r and asking it to explain why the relation holds. MSLM then masks \"relation spans\" with 0.8 probability and SCL contrasts positive spans like \"was born in\" against negative spans like \"New York,\" but the paper never states how these spans are extracted or whether the LLM-generated explanation is used to obtain them. If relation spans are defined by matching the relation name or description inside the explanation, MSLM is reconstructing tokens that carry the supervised label, and SCL is pulling span embeddings toward an anchor derived from the same label. In that case, the pre-training losses become a supervised relation-classification signal in disguise, and the reported F1 gains over BERT-style masking in Table 3 may reflect label leakage or surface cues rather than \"relational reasoning.\" Eq. (3) assigns masking probabilities by \"linguistic role\" but supplies no role-labeling procedure; no hyperparameters (lambda_1, lambda_2, tau) or span extraction algorithm are given anywhere in the paper or appendix. This makes the central mechanism unfalsifiable as written, although the released code and data could resolve it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TKRE, a two-stage framework for few-shot relation extraction. In the first stage, a frozen LLM generates relation explanations conditioned on the golden relation label and schema-constrained synthetic instances. In the second stage, a small discriminative model (TYP Marker or GenPT) is pre-trained with Masked Span Language Modeling (MSLM) and Span-Level Contrastive Learning (SCL), then fine-tuned on a mixture of golden and synthetic examples. Experiments on SemEval, TACRED, TACREV, and Re-TACRED under 8/16/32-shot settings report state-of-the-art F1, including average gains of 7.8 points over TYP Marker and 5.0 points over GenPT. Ablations attribute the gains to the knowledge-guided pre-training components, and additional analysis examines sensitivity to the LLM used for explanation generation and to the volume of synthetic data.","tokens_in":17277,"tokens_out":5891,"duration_ms":57233,"significance":"If the central claim is correct, TKRE provides a practical recipe for transferring generative LLM knowledge into compact discriminative relation-extraction models, and the two-stage pre-training objectives would be a reusable contribution. The paper has clear strengths: it releases code and data, evaluates on four standard datasets under multiple few-shot settings, ablates every proposed component, and shows robustness across three different LLM generators. However, the manuscript does not specify the span-selection procedure that defines MSLM/SCL, does not report variance or statistical significance, and does not establish that all baselines are compared under identical backbones and training conditions. These gaps must be closed before the paper's central claim about the pre-training objectives can be accepted.","major_comments":[{"comment":"The central pre-training mechanism is not specified operationally. Eq. (3) assigns masking probabilities by 'linguistic role' (relation span, subject/object entity span, other), and Eq. (6) requires identifying positive spans such as 'was born in' and negative spans such as 'New York', but the manuscript never states how these spans are extracted from the LLM-generated corpus. Because the explanations in Eq. (2) are generated with the golden relation label r in the prompt, an unspecified span-selection rule could make MSLM reconstruct label-carrying tokens and SCL use an anchor derived from the same label, effectively turning the pre-training losses into a supervised relation-classification signal. Please provide the exact span-extraction algorithm (or the parsing rules and prompts used), give the hyperparameter values for lambda_1, lambda_2, and tau, and report an ablation with random spans to rule out label leakage.","section":"Sec. 3.3, Eqs. (3)-(6)"},{"comment":"The prose in this section repeatedly describes entity recognition rather than relation extraction: it says the instruction asks the LLM to explain 'why a given text span should be recognized as an entity' and mentions 'domains relevant to NER tasks.' As written, this does not describe how the relation explanation corpus C is built, even though Eq. (2), Figure 3, and the downstream pre-training all depend on relation-level explanations. Please rewrite this section to specify the exact instruction template for relation explanations and the post-processing steps used to construct C.","section":"Sec. 3.2, Explanation-Driven Knowledge Generation"},{"comment":"All reported F1 scores appear to come from a single run with no standard deviation or significance test. Several key comparisons are small in magnitude (e.g., Table 3: w/oSCL is 57.9 vs. 59.6 for TKRE on the averaged columns; w/oMSLM is 57.2), yet the text repeatedly uses the word 'significant.' Please report means and standard deviations over at least three seeds with different random few-shot splits, and, ideally, paired significance tests for the main comparisons.","section":"Sec. 4.2 and 4.3, Tables 2 and 3"},{"comment":"The baseline comparison may not be apples-to-apples. The appendix states that RoBERTa-large is the base architecture for the authors' TYP Marker and GenPT implementations, but it does not state whether CP, KnowPrompt, PTRE, Unleash, and S2ynRE were re-run under the same backbone, hyperparameters, and training budgets. If those numbers are quoted from the original papers, differences in backbone or tuning could account for part of the reported gains. Please clarify this explicitly, and ideally re-run the key baselines under the same settings as TKRE.","section":"Appendix A.2, Table 2"}],"minor_comments":[{"comment":"The table caption says 'Ablation study results for TACRED' but the table reports columns for both SemEval and TACRED; please correct the caption or reorganize the table.","section":"Table 3"},{"comment":"The column layout is confusing: the header includes '8-Shot 16-Shot 32-Shot All' but the rows then list numbers such as 148, 295, 589, 6,507, 1,439, 2,717, and 19. Please clarify what each number denotes and split the few-shot statistics from the full-dataset statistics.","section":"Table 1"},{"comment":"The sentence 'However, it exhibits the generation performance in FSRE' appears to be missing a qualifier; it should likely read 'poor generation performance' or 'suboptimal generation performance.'","section":"Sec. 4.2, observation 2"},{"comment":"The symbol gamma is defined as the maximum input sequence length but is used as if it were the number of masked tokens; please make the notation consistent.","section":"Eq. (5)"},{"comment":"The line labeled 'RE-TACRED (Supervised w/ Golden)' is not explained in the caption or text; please state what this baseline represents and how it was computed.","section":"Fig. 6"},{"comment":"The description of Unleash duplicates the description given for GenPT/KnowPrompt; please replace it with an accurate summary of Unleash's method.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to contain leftover text from a prior NER-oriented paper in Section 3.2, and the reporting of experiments and span-selection procedures is incomplete. The released code and data may resolve the span-extraction concern, but the write-up as submitted is not yet reproducible. The overall idea is plausible and worth pursuing, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a competent empirical paper that likely delivers on its main claim — combining LLM-generated explanations and synthetic data with span masking and contrastive learning improves few-shot relation extraction — but it leaves the most interesting part of the method under-specified. The gains (7.8% over TYP Marker, 5.0% over GenPT) are real on four datasets, and the ablations suggest every component contributes. The code and data are released, which is a plus.\n\nWhat's new: the specific integration of explanation-driven knowledge and schema-constrained synthetic data into a two-stage pre-training objective (MSLM + SCL) is not in the literature. The components are known, but the combination is new, and the experiments are reasonably thorough.\n\nThe soft spots, in order of severity. First, the span extraction procedure for MSLM and SCL is never described. Section 3.3 gives masking probabilities (0.8/0.5/0.2 for relation/entity/other spans) but not how relation spans are identified. The explanation corpus is built by prompting an LLM with the golden relation label; if \"relation spans\" are simply the phrases in the explanation that mention the relation, then MSLM is reconstructing tokens that carry the label, and SCL pulls span embeddings toward an anchor derived from the same label. That is not necessarily a flaw — it is a form of knowledge distillation — but the paper does not state it. As written, the central pre-training contribution is unfalsifiable, and the reported gains could be due to surface cues rather than relational reasoning. The released code probably resolves this, but the paper needs to say how.\n\nSecond, the baseline numbers may not be strictly comparable. The paper says it reruns TYP Marker and GenPT on RoBERTa-large, but the other baselines (CP, KnowPrompt, PTRE) are taken from their original papers, which may have used different backbones or settings. A single standard evaluation protocol would make the comparison cleaner.\n\nThird, all results are single-run with no variance or significance tests. The differences are large enough that they would likely survive, but we can't know.\n\nMinor: calling this \"pre-training\" is a stretch — it's additional training on a tiny augmented corpus. The synthetic data scale is only double the K-shot set, so it's not large-scale pre-training.\n\nOverall, the paper deserves a serious referee. The empirical claim is valuable and the code is available, but the authors need to specify the span extraction, report variance, and either harmonize baselines or justify why they didn't. If I were the editor, I'd send it out, expecting major revisions on the methodology description.\n\nI'd bring it to a reading group only if the group cares about few-shot IE; otherwise it's a \"maybe.\" I'd cite it if I worked in that space — the combination is useful to know.\n\nIn short: send it to review, but flag the span-selection question as the key issue.","headline":"TKRE is a solid, incremental FSRE paper with real but under-specified gains; send it to review but demand the span-extraction details.","tokens_in":17814,"tokens_out":3488,"would_cite":true,"duration_ms":32694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TKRE, a two-stage knowledge-guided pre-training framework, converts LLM-generated explanations and synthetic data into span-masking and contrastive learning signals, and reports state-of-the-art few-shot relation extraction results, with…","keywords":["few-shot relation extraction","large language models","synthetic data","masked span language modeling","span-level contrastive learning","knowledge-guided pre-training"],"falsifier":"Run TKRE with the relation-indicative span annotation replaced by random spans of matched length, keeping all other components fixed; if the F1 advantage over GenPT and TYP Marker disappears, the gains are caused by the span-identification assumption rather than by the MSLM/SCL objectives themselves.","tokens_in":16811,"feed_emoji":"🤖","tokens_out":6989,"duration_ms":57312,"temperature":0.7,"pith_summary":"The paper argues that the gap between generative and discriminative learning can be bridged for few-shot relation extraction by converting LLM knowledge into supervised pre-training signals. Specifically, it proposes TKRE, which prompts an LLM to produce relation explanations and schema-constrained synthetic examples, then pre-trains a small relation-extraction model with two auxiliary tasks: masked span language modeling, which reconstructs relation-indicative spans, and span-level contrastive learning, which pulls relation-relevant spans together and pushes irrelevant ones apart. On SemEval, TACRED, TACREV, and Re-TACRED, the method reports F1 gains of 7.8 points over TYP Marker and 5.0 points over GenPT, and it outperforms both direct LLM prompting and existing hybrid methods. If correct, this shows that the structured knowledge of LLMs can be transferred to compact discriminative models without fine-tuning the LLM itself.","feed_headline":"LLM knowledge lifts few-shot relation extraction by 7.8 points","feed_subtitle":"Small models trained with LLM explanations outperform direct GPT-4 prompting in few-shot relation extraction.","key_machinery":"The load-bearing mechanism is the two-stage pre-training loss that combines Masked Span Language Modeling and Span-level Contrastive Learning. MSLM extends BERT-style masking to continuous spans, with a token-type-dependent masking probability that makes relation-indicative spans the primary reconstruction target, teaching the model which lexical cues signal a relation. SCL sharpens the resulting span representations by contrasting positive spans that express the relation with negative spans that are contextually plausible but relationally wrong. The synthetic data and explanation corpus are the raw material that makes these objectives effective; without the LLM-generated knowledge, the span-level tasks have no relation-specific signal to learn from.","core_discovery":"The central claim is that a small discriminative relation-extraction model can be made to learn the relational logic that LLMs encode by exposing it to two kinds of LLM-generated signal—free-text explanations of why an entity pair holds a relation, and schema-constrained pseudo-instances—and then training it with two span-level objectives. The first objective, MSLM, masks continuous spans with probabilities tuned by syntactic role (relation spans are masked with 0.8 probability, entity spans with 0.5, and other words with 0.2), forcing the model to reconstruct the relational words that carry the semantics. The second, SCL, uses a contrastive loss that pulls the embedding of a relation-relevant span (e.g., \"was born in\") toward the relation-type anchor and pushes a relation-irrelevant span (e.g., \"New York\") away. Together with fine-tuning on the few golden examples plus the synthetic data, this pipeline yields the reported state-of-the-art F1 scores on four benchmark datasets.","pith_inferences":["The same recipe could transfer to other structured prediction tasks (e.g., named entity recognition, event extraction) where LLMs can verbalize the decision rule and a small model can be pre-trained to reconstruct and contrast the spans that embody that rule.","The paper does not specify how relation-indicative and positive/negative spans are identified in the LLM-generated corpus; one testable extension is to replace the manual span annotation with attention-based selection from the small model itself, which would tell us whether the gains come from the annotation or from the objectives.","The observed decline with excessive synthetic data suggests that adding a quality filter or confidence threshold on LLM-generated instances could push the performance frontier further, a check that the paper does not run.","If the method truly transfers knowledge without leaking relation labels through span selection, it should also work when the LLM explanations are generated from a different domain or language than the target dataset, which would be a strong test of the 'relational logic' hypothesis."],"forward_implications":["TKRE's reported F1 gains of 7.8 points over TYP Marker and 5.0 points over GenPT imply that LLM knowledge can be transferred to small relation-extraction models without parameter-heavy fine-tuning of the LLM itself.","The framework outperforms pure LLM in-context learning (GPT-4, GPT-3.5, LLaMA-2) in few-shot settings, suggesting that discriminative models with LLM-guided pre-training are a more sample-efficient route to low-resource relation extraction than relying on the generative model alone.","Ablations show that both the explanation corpus and the schema-constrained synthetic data contribute, and that replacing MSLM with BERT-style random token masking or removing SCL degrades performance, so the span-level objectives are not redundant.","The best performance is achieved at a moderate amount of synthetic data; adding more generated data beyond a threshold degrades results, so the method's gains depend on controlling synthetic-data noise."],"supporting_citations":[{"why":"Supplies the TYP Marker baseline that TKRE improves by 7.8 F1 points.","marker":"[Zhou and Chen, 2022]"},{"why":"Supplies the GenPT baseline that TKRE improves by 5.0 F1 points.","marker":"[Han et al., 2022]"},{"why":"Introduces the schema-constrained LLM data augmentation approach that TKRE builds on and compares against as Unleash.","marker":"[Xu et al., 2023b]"},{"why":"Presents a two-stage self-training hybrid baseline (S2ynRE) that TKRE outperforms.","marker":"[Xu et al., 2023a]"},{"why":"Motivates the contrastive learning objective used in SCL for relation-aware span representations.","marker":"[Qin et al., 2021]"},{"why":"Provides the original masked language modeling objective that MSLM extends to span-level masking.","marker":"[Devlin et al., 2019]"},{"why":"RoBERTa-large is the base architecture for the relation extraction models in TKRE.","marker":"[Liu et al., 2019]"},{"why":"Provides the TACREV dataset and documents the confusion of semantically similar relations that motivates span-level contrastive learning.","marker":"[Alt et al., 2020]"},{"why":"Provides the Re-TACRED dataset used in evaluation and explains its cleaner labels.","marker":"[Stoica et al., 2021]"},{"why":"Provides the TACRED dataset that serves as a primary benchmark in the experiments.","marker":"[Zhang et al., 2017]"}],"fun_headline_variants":["LLM explanations train small RE models to beat direct GPT-4 prompting","Two-stage pre-training: LLM knowledge lifts few-shot relation extraction","TKRE uses LLM insights to surpass direct LLM prompting in RE","Small discriminative model + LLM knowledge wins few-shot relation extraction","LLM-generated data and span-level contrastive learning boost FSRE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the spans marked as relation-indicative and as positive/negative contrastive examples can be identified reliably in the LLM-generated corpus; the paper never specifies the extraction or annotation procedure, so if that identification is noisy or leaks the relation label, the reported gains could come from the synthetic data rather than from the pre-training objectives.","fun_headline_variants_meta":{"raw":{"variants":["LLM explanations train small RE models to beat direct GPT-4 prompting","Two-stage pre-training: LLM knowledge lifts few-shot relation extraction","TKRE uses LLM insights to surpass direct LLM prompting in RE","Small discriminative model + LLM knowledge wins few-shot relation extraction","LLM-generated data and span-level contrastive learning boost FSRE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1755,"prompt_tokens":987,"completion_tokens":768,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":675}},"tokens_in":603,"tokens_out":768,"duration_ms":7365,"temperature":1.0,"reasoning_tokens":675,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:37:29.380323+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TKRE with the relation-indicative span annotation replaced by random spans of matched length, keeping all other components fixed; if the F1 advantage over GenPT and TYP Marker disappears, the gains are caused by the span-identification assumption rather than by the MSLM/SCL objectives themselves.","supporting_citations":[{"cited_title":"Position-aware attention and supervised data improve slot filling","cited_arxiv_id":null,"evidence_quote":"Provides the TACRED dataset that serves as a primary benchmark in the experiments."},{"cited_title":"An improved baseline for sentence-level relation extraction","cited_arxiv_id":null,"evidence_quote":"Supplies the TYP Marker baseline that TKRE improves by 7.8 F1 points."},{"cited_title":"Erica: Improving entity and relation understanding for pre-trained language models via contrastive learning","cited_arxiv_id":null,"evidence_quote":"Motivates the contrastive learning objective used in SCL for relation-aware span representations."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Provides the original masked language modeling objective that MSLM extends to span-level masking."},{"cited_title":"Tacred revisited: A thorough evaluation of the tacred relation extraction task","cited_arxiv_id":null,"evidence_quote":"Provides the TACREV dataset and documents the confusion of semantically similar relations that motivates span-level contrastive learning."},{"cited_title":"Re-tacred: Addressing shortcomings of the tacred dataset","cited_arxiv_id":null,"evidence_quote":"Provides the Re-TACRED dataset used in evaluation and explains its cleaner labels."}],"review_version":1}