{"id":"6f23a57e-de69-4a23-8a3e-9e7b5e959b9a","arxiv_id":"2412.05434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Holding training set size fixed, increasing the number of relation types improves few-shot relation classification performance on unseen relations, demonstrated in the new REBEL-FS benchmark.","lead":"Researchers built a benchmark with over 900 relation types and found that models trained on a fixed number of examples spread across many relation types generalize better to unseen relations than models trained on the same number of examples from few relation types. The finding suggests data diversity can substitute for data quantity in few-shot relation classification, potentially reducing annotation costs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's diversity gains may stem from easier negative pairs rather than broader relation coverage; more relation types lower average negative-pair similarity, which can raise threshold-based F1 even without better ranking.","rationale":"The reader's weakest assumption identifies the same core threat: the diversity manipulation is not causally isolated because it changes the composition of negative pairs. I agree with that diagnosis and with the CONDITIONAL verdict. My stress-test sharpens the mechanism: the Siamese model is evaluated with F1, a threshold-dependent metric, so a shift in the negative-pair similarity distribution can inflate F1 without any improvement in the model's underlying ranking ability. The paper does not report AUC or average precision, nor does it measure negative-pair difficulty, so this alternative explanation is live. The fix is concrete and inexpensive: re-analyze the existing model outputs with ranking metrics and, if needed, rebalance negative-pair difficulty across conditions. The paper's broader evidence (cross-dataset M-way K-shot results, learning curves) suggests the diversity hypothesis may well be correct, but Table 2 as presented does not yet isolate it. Hence the verdict remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":11474,"tokens_out":7221,"duration_ms":77623,"concrete_test":"Recompute Table 2 from the trained Siamese models using threshold-free ranking metrics (ROC-AUC or average precision) on the identical fixed test sets. If AUC rises with relation count, the diversity effect reflects genuinely better separation; if AUC is flat or lower while F1 rises, the reported gains are an artifact of the negative-pair similarity shift and the chosen threshold. Additionally, for each condition, compute the mean cosine similarity of the negative training pairs; if this quantity falls sharply as relation count increases, match negative-pair difficulty across conditions (e.g., sample negative pairs with equal mean similarity) and verify the F1 trend persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the controlled comparison that carries the paper's central claim (Section 5.5, Table 2), the number of training relation types is varied from 29 to 576 at fixed 100K examples, but the source pool for negative pairs changes with it. The 29-relation condition contains only relations with at least 5000 instances, which are the most frequent and likely the most semantically confusable; the 576-relation condition adds rare relations, so negative pairs are sampled from a far more heterogeneous set. This changes the marginal distribution of negative-pair similarities: with more relations, a randomly sampled negative pair is typically less similar and easier to reject. Since F1 is computed from a threshold on cosine similarity, a leftward shift in the negative score distribution raises precision and F1 even if the model's ability to rank positive above negative pairs is unchanged. The paper reports no threshold-independent metric (AUC or average precision) and no measure of negative-pair difficulty across conditions. The nested construction also confounds relation count with relation frequency and per-relation example count: high-diversity conditions include rarer relations and fewer examples per relation, so the effect cannot be uniquely attributed to 'diversity' as opposed to the composition of the selected relation set. Without controlling for negative-pair difficulty, Table 2 does not establish that broader relation coverage, rather than an easier training task, drives the gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that in few-shot relation classification (FSRC), the diversity of relation types in training data matters more than raw data quantity. It introduces REBEL-FS, a benchmark derived from the REBEL dataset with 954 relation types, and reports experiments in which the total number of training examples is held fixed while the number of training relation types is varied (29 to 576 in the main ablation, Table 2). The authors find that higher relation diversity improves F1 on relation-pair classification across several FSRC test sets, especially under high-negative settings, and also report that small but diverse training sets can match larger ones. The central claim is that targeted data curation focusing on relation diversity can substantially reduce the need for large-scale datasets in FSRC.","tokens_in":11735,"tokens_out":5179,"duration_ms":51086,"significance":"If the diversity effect is causally isolated, the result is practically important for dataset construction and few-shot learning: it suggests a cheap intervention, adding more relation types at a fixed example budget, improves generalization, and the REBEL-FS benchmark with 576 training relations is a potentially useful resource. The paper includes controlled experiments with fixed total examples, multiple test sets, and a clear statement of the diversity hypothesis. However, the main ablation does not currently isolate diversity from changes in negative-pair difficulty, so the strength of the causal claim is not yet established. The benchmark itself and the systematic comparison across relation counts are useful contributions even if the causal interpretation requires further controls.","major_comments":[{"comment":"The diversity manipulation is confounded with negative-pair composition. Increasing the number of relation types from 29 to 576 is implemented by lowering the minimum frequency threshold (M from 5000 to 40), so the high-diversity conditions include rarer, more heterogeneous relations. At fixed 100K training examples, negative pairs (different relation types) are then drawn from a larger and more semantically separated pool, shifting the negative cosine-similarity distribution left. Since the reported F1 is computed by thresholding cosine similarity, an easier negative-pair distribution raises precision and F1 even if the model's ranking quality is unchanged. The paper reports no threshold-independent metric (e.g., AUC or average precision) and no measure of negative-pair difficulty across conditions. The nested construction also changes per-relation example count (100K/29 vs 100K/576) and relation frequency, so the effect cannot be uniquely attributed to 'diversity.' This is load-bearing for the central claim. Please add AUC/AP, report negative-pair similarity distributions, and include a control that keeps negative-pair composition fixed across relation counts, such as evaluating on a common set of negative-pair relation pairs.","section":"Section 5.5, Table 2"},{"comment":"The introduction reports 'up to 157%' improvement in high-negative settings and cites 91.3% versus 80.9% for models trained on 400 relation types. Table 2 contains no 157% value; the largest proportional gain at 99% negatives is 122.5% (14.22 vs 6.39), and the relation counts in Table 2 are 29, 79, 233, 461, and 576, not 400. The 91.36 F1 corresponds to 576 relation types, not 400. Please align the claims with the reported tables and remove or justify the 157% figure.","section":"Section 1 vs Section 5.5, Table 2"},{"comment":"The paper states that bold results are statistically significant (p < 0.05) and underlined results are highest but not significant, yet Table 2 and the surrounding text do not mark significance or report variance despite Section 5.3 saying models were trained and evaluated three times. Without confidence intervals or a significance test, differences between adjacent rows (e.g., 461 vs 576 relation types at 90% negatives: 66.48 vs 66.52) cannot be assessed. Please report per-condition variance and apply the stated significance testing to Table 2.","section":"Section 5.2.1 and Table 2"}],"minor_comments":[{"comment":"Typos: 'farther explain' should be 'further explain' and 'summerize' should be 'summarize'.","section":"Section 1"},{"comment":"The table header 'RECESS-FS' appears to be a typo for 'REBEL-FS'.","section":"Appendix A, Table 3"},{"comment":"The Siamese evaluation does not specify how the cosine-similarity threshold is chosen; since F1 depends on the threshold, please state whether it is tuned on a development set and report the threshold or the threshold selection procedure.","section":"Section 5.2.2"},{"comment":"The reference list contains duplicated FewRel 2.0 entries (Gao et al., 2019 and Gao et al., 2020) and several incompletely formatted entries (e.g., the Phi-3 technical report and 'Textbooks Are All You Need'); please clean up the bibliography.","section":"References"},{"comment":"Section 5.1 says relation types are varied as 10, 50, 100, 200, 400, while Table 2 uses 29, 79, 233, 461, 576; this inconsistency should be resolved.","section":"Section 5.1 and Table 2"},{"comment":"The caption for Figure 4 does not explain the dashed lines; the text does, but please make the caption self-contained and note that the reported metric is accuracy.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the benchmark is a useful resource, but the central claim requires the negative-pair difficulty control described in the major comments. The reported gains in high-negative settings are large and interesting, which makes the confound particularly important to resolve. I would recommend sending the revised version back to the same reviewers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful empirical paper with a real new resource (REBEL-FS) and a plausible central claim, but the headline causal story is not fully nailed down. The confound between relation count and relation frequency, plus missing variance and a numeric inconsistency in the intro, means the paper needs revision before the claim is airtight.\n\nThe strength is the systematic ablation: fixing total examples and varying the number of relation types from 29 to 576, they see F1 rise from 80.9 to 91.36 in the balanced case and larger relative gains in high-negative settings. That pattern holds across several test sets, and the learning-curve analysis showing less overfitting with more relation types is a nice supporting observation. The authors also deserve credit for building REBEL-FS from REBEL, which is a genuine contribution: an order of magnitude more relation types than existing FSRC benchmarks, and they make the data construction transparent.\n\nWhere it gets soft: the diversity manipulation is not clean. The 29-relation condition uses relations with at least 5000 instances, while the 576-condition includes down to 40 instances. So relation count is tangled with per-relation frequency and with the distribution of negative pairs in training. That means the effect could be partly due to seeing a wider spread of relation types, which is the hypothesis, but also partly due to having a more heterogeneous negative-pair pool that makes the training task easier. A threshold-independent metric like AUC or average precision would help, and a control that varies negative-pair difficulty while holding relation count fixed would strengthen the attribution. The stress-test note is right to raise this; it doesn't sink the paper, but it should be addressed.\n\nAlso, the intro's claim of \"up to 157%\" doesn't match Table 2's max of 122.5%. And Table 2 lacks error bars, even though the setup says runs were averaged. These are fixable but should be fixed.\n\nOverall, this is a useful paper for anyone working on few-shot relation classification or dataset curation. It deserves a serious referee, with the request to address the confound and report more careful numbers.","headline":"A useful empirical study with a new benchmark and a plausible diversity-over-quantity claim, but the central causal attribution is weakened by a relation-count/frequency confound and sloppy reported numbers.","tokens_in":12268,"tokens_out":2986,"would_cite":true,"duration_ms":30616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Relation type diversity—not example count—is what lets few-shot classifiers generalize to unseen relations.","keywords":["few-shot relation classification","relation diversity","REBEL-FS","data curation","few-shot generalization","None-of-the-Above","siamese network","overfitting"],"falsifier":"Train two models on the same 576 relations and same sentence pairs, but in one condition randomly permute relation labels so that the diversity count is unchanged while semantic distinctness is destroyed; if the F1-versus-relation-type sweep still climbs, the effect is not semantic diversity. Alternatively, keep 29 relations and construct negative pairs with the same average pairwise similarity as the 576-relation condition; if F1 jumps to the 576-relation level, the paper's manipulation is confounded by negative-pair difficulty.","tokens_in":11276,"feed_emoji":"📈","tokens_out":7502,"duration_ms":69204,"temperature":0.7,"pith_summary":"This paper argues that in few-shot relation classification (FSRC), the diversity of relation types in training data matters more than the number of training examples. The authors introduce REBEL-FS, a benchmark built from the REBEL corpus with 954 relation types (576 for training), and hold total example counts fixed while varying how many relation types a model sees. Across their controlled sweeps, F1 rises from 80.9 with 29 relation types to 91.36 with 576 relation types at 100K examples, and the relative gains grow as the share of negative examples increases. A model trained on just 1K diverse examples (90.75 F1) nearly matches a 100K-example model, which the paper takes as evidence that targeted curation of diverse relations can substitute for large-scale data.","feed_headline":"Relation diversity, not dataset size, lifts few-shot classification","feed_subtitle":"576 relation types reach 91.36 F1 where 29 reach 80.9; 1K diverse examples nearly match 100K.","key_machinery":"Three components carry the argument. REBEL-FS, the new dataset, is the central object: built from the 1,146-relation REBEL corpus, it is split by frequency into 576 training, 100 development, and 278 test relations, with examples constructed by pairing sentences that share a relation (positive) or do not (negative). The relation-specific evaluation, a siamese binary setup, replaces M-way K-shot with a single same-relation/different-relation decision, which lets the paper vary the NOTA ratio (50%, 10%, and 1% positives) cleanly. The model is a siamese network over SpanBERT span-marker representations, trained with an L2 objective on cosine similarity; the controlled sweeps vary relation-type count, negative ratio, and training size against this fixed machinery.","core_discovery":"On the paper's own terms, the central discovery is that generalization to novel relations in FSRC is driven primarily by the variety of relation types seen during training, not by dataset size. The authors demonstrate this by sweeping the number of training relation types from 29 to 576 while holding the example budget at 100K: the relation-specific binary-evaluation F1 climbs from 80.9 to 91.36, and in a 99%-negative setting it climbs from 6.39 to 14.22. The same sweep with 576 relation types but only 1K training examples yields 90.75 F1, within half a point of the full-data model, so small-but-diverse data reproduces most of the large-data result. They interpret the relation space as a spanning set: a model that has seen many distinct relation structures has already acquired the linguistic and semantic components needed for novel relations, and this shows up as stable training (no observable overfitting for 461- and 576-relation models) and robust performance when negatives dominate.","pith_inferences":["If the slope in F1 versus relation-type count is driven by coverage, the same curation principle should transfer to other few-shot tasks whose labels carve a semantic space (event types, intents, property types); the paper only demonstrates it for relations.","A direct test of the causal story would hold negative-pair difficulty fixed while varying relation-type count; the paper reports no such control, so a skeptic can still explain the sweep as an easier decision problem rather than richer semantic coverage.","Because REBEL-FS is silver-standard, the results imply that noisy but broad supervision can outperform cleaner narrow supervision; a controlled comparison with a human-annotated subset would make this explicit.","The learning-curve result suggests diversity acts as a regularizer, which could be probed by measuring representational geometry (for example, separation of relation clusters) rather than only F1."],"forward_implications":["More diverse relation-type coverage reliably raises F1 on unseen relations at fixed example counts: 80.9 to 91.36 F1 at 100K examples when relation types grow from 29 to 576.","High-negative settings benefit most: at 90% negatives F1 rises from 41.12 to 66.52, and at 99% negatives from 6.39 to 14.22 as diversity increases.","A small diverse dataset can substitute for a large one: 1K examples with 576 relation types reach 90.75 F1 versus 91.36 for 100K examples at 50% negatives.","Diverse training mitigates overfitting: models trained on 29 and 79 relation types overfit within the first epoch, while 461- and 576-relation models show no observable overfitting across four epochs."],"supporting_citations":[{"why":"Supplies REBEL, the 1,146-relation, nine-million-instance corpus from which REBEL-FS is built.","marker":"(Huguet Cabot and Navigli, 2021)"},{"why":"Supplies FewRel 2.0, the 100-relation Wikipedia benchmark with NOTA examples used for cross-dataset evaluation.","marker":"(Gao et al., 2019)"},{"why":"Supplies CORE, the 12-relation company dataset used as a test set in generalization experiments.","marker":"(Borchert et al., 2023)"},{"why":"Supplies TACRED-FS, the 41-relation high-NOTA benchmark used both as a comparison and to probe domain shift.","marker":"(Sabo et al., 2021a)"},{"why":"Provides SpanBERT, whose span-marker representations feed the siamese relation-specific model.","marker":"(Joshi et al., 2020)"},{"why":"Provides BERT, used as the base model for the M-way K-shot evaluation for consistency with prior work.","marker":"(Devlin et al., 2018)"},{"why":"Establishes siamese BERT sentence embeddings, the architecture adapted for the relation-specific setup.","marker":"(Reimers and Gurevych, 2020)"},{"why":"Motivates the siamese network with metric learning used to train the diversity sweeps.","marker":"(Schroff et al., 2015)"}],"fun_headline_variants":["Diverse relations beat big data in few-shot classification","576 relation types beat 29: small diverse data wins","Few-shot: diversity beats dataset size","Relation diversity gives 91 F1 with 1K examples","Small diverse data matches 100K in few-shot relation task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments hold the number of training examples fixed while changing the number of relation types, and this also changes which relation pairs count as negatives; the paper does not control for whether those negative pairs simply become easier to tell apart, so the causal role of diversity is not fully isolated.","fun_headline_variants_meta":{"raw":{"variants":["Diverse relations beat big data in few-shot classification","576 relation types beat 29: small diverse data wins","Few-shot: diversity beats dataset size","Relation diversity gives 91 F1 with 1K examples","Small diverse data matches 100K in few-shot relation task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2180,"prompt_tokens":921,"completion_tokens":1259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1181}},"tokens_in":537,"tokens_out":1259,"duration_ms":8650,"temperature":1.0,"reasoning_tokens":1181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:43:13.049140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two models on the same 576 relations and same sentence pairs, but in one condition randomly permute relation labels so that the diversity count is unchanged while semantic distinctness is destroyed; if the F1-versus-relation-type sweep still climbs, the effect is not semantic diversity. Alternatively, keep 29 relations and construct negative pairs with the same average pairwise similarity as the 576-relation condition; if F1 jumps to the 576-relation level, the paper's manipulation is confounded by negative-pair difficulty.","supporting_citations":[],"review_version":1}