REVIEW 3 major objections 6 minor 27 references
Diversity Over Quantity: A Lesson From Few Shot Relation Classification
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Relation type diversity—not example count—is what lets few-shot classifiers generalize to unseen relations.
desk verdict A useful empirical study with a new benchmark and a plausible diversity-over-quantity claim, but the central causal attribution is weakened by a relation-count/frequency confound and sloppy reported numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three components carry the argument. REBEL-FS, the new dataset, is the central object: built from the 1,146-relation REBEL corpus, it is split by frequency into 576 training, 100 development, and 278 test relations, with examples constructed by pairing sentences that share a relation (positive) or do not (negative). The relation-specific evaluation, a siamese binary setup, replaces M-way K-shot with a single same-relation/different-relation decision, which lets the paper vary the NOTA ratio (50%, 10%, and 1% positives) cleanly. The model is a siamese network over SpanBERT span-marker representations, trained with an L2 objective on cosine similarity; the controlled sweeps vary relation-type count, negative ratio, and training size against this fixed machinery.
What would settle it
Train two models on the same 576 relations and same sentence pairs, but in one condition randomly permute relation labels so that the diversity count is unchanged while semantic distinctness is destroyed; if the F1-versus-relation-type sweep still climbs, the effect is not semantic diversity. Alternatively, keep 29 relations and construct negative pairs with the same average pairwise similarity as the 576-relation condition; if F1 jumps to the 576-relation level, the paper's manipulation is confounded by negative-pair difficulty.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that generalization to novel relations in FSRC is driven primarily by the variety of relation types seen during training, not by dataset size. The authors demonstrate this by sweeping the number of training relation types from 29 to 576 while holding the example budget at 100K: the relation-specific binary-evaluation F1 climbs from 80.9 to 91.36, and in a 99%-negative setting it climbs from 6.39 to 14.22. The same sweep with 576 relation types but only 1K training examples yields 90.75 F1, within half a point of the full-data model, so small-but-diverse data reproduces most of the large-data result. They interpret the relation space as a spanning set: a model that has seen many distinct relation structures has already acquired the linguistic and semantic components needed for novel relations, and this shows up as stable training (no observable overfitting for 461- and 576-relation models) and robust performance when negatives dominate.
Load-bearing premise
The experiments hold the number of training examples fixed while changing the number of relation types, and this also changes which relation pairs count as negatives; the paper does not control for whether those negative pairs simply become easier to tell apart, so the causal role of diversity is not fully isolated.
Editorial extensions
If this is right
- More diverse relation-type coverage reliably raises F1 on unseen relations at fixed example counts: 80.9 to 91.36 F1 at 100K examples when relation types grow from 29 to 576.
- High-negative settings benefit most: at 90% negatives F1 rises from 41.12 to 66.52, and at 99% negatives from 6.39 to 14.22 as diversity increases.
- A small diverse dataset can substitute for a large one: 1K examples with 576 relation types reach 90.75 F1 versus 91.36 for 100K examples at 50% negatives.
- Diverse training mitigates overfitting: models trained on 29 and 79 relation types overfit within the first epoch, while 461- and 576-relation models show no observable overfitting across four epochs.
Reading between the lines
- If the slope in F1 versus relation-type count is driven by coverage, the same curation principle should transfer to other few-shot tasks whose labels carve a semantic space (event types, intents, property types); the paper only demonstrates it for relations.
- A direct test of the causal story would hold negative-pair difficulty fixed while varying relation-type count; the paper reports no such control, so a skeptic can still explain the sweep as an easier decision problem rather than richer semantic coverage.
- Because REBEL-FS is silver-standard, the results imply that noisy but broad supervision can outperform cleaner narrow supervision; a controlled comparison with a human-annotated subset would make this explicit.
- The learning-curve result suggests diversity acts as a regularizer, which could be probed by measuring representational geometry (for example, separation of relation clusters) rather than only F1.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that in few-shot relation classification (FSRC), the diversity of relation types in training data matters more than raw data quantity. It introduces REBEL-FS, a benchmark derived from the REBEL dataset with 954 relation types, and reports experiments in which the total number of training examples is held fixed while the number of training relation types is varied (29 to 576 in the main ablation, Table 2). The authors find that higher relation diversity improves F1 on relation-pair classification across several FSRC test sets, especially under high-negative settings, and also report that small but diverse training sets can match larger ones. The central claim is that targeted data curation focusing on relation diversity can substantially reduce the need for large-scale datasets in FSRC.
Significance. If the diversity effect is causally isolated, the result is practically important for dataset construction and few-shot learning: it suggests a cheap intervention, adding more relation types at a fixed example budget, improves generalization, and the REBEL-FS benchmark with 576 training relations is a potentially useful resource. The paper includes controlled experiments with fixed total examples, multiple test sets, and a clear statement of the diversity hypothesis. However, the main ablation does not currently isolate diversity from changes in negative-pair difficulty, so the strength of the causal claim is not yet established. The benchmark itself and the systematic comparison across relation counts are useful contributions even if the causal interpretation requires further controls.
major comments (3)
- [Section 5.5, Table 2] The diversity manipulation is confounded with negative-pair composition. Increasing the number of relation types from 29 to 576 is implemented by lowering the minimum frequency threshold (M from 5000 to 40), so the high-diversity conditions include rarer, more heterogeneous relations. At fixed 100K training examples, negative pairs (different relation types) are then drawn from a larger and more semantically separated pool, shifting the negative cosine-similarity distribution left. Since the reported F1 is computed by thresholding cosine similarity, an easier negative-pair distribution raises precision and F1 even if the model's ranking quality is unchanged. The paper reports no threshold-independent metric (e.g., AUC or average precision) and no measure of negative-pair difficulty across conditions. The nested construction also changes per-relation example count (100K/29 vs 100K/576) and relation frequency, so the effect cannot be uniquely attributed to 'diversity.' This is load-bearing for the central claim. Please add AUC/AP, report negative-pair similarity distributions, and include a control that keeps negative-pair composition fixed across relation counts, such as evaluating on a common set of negative-pair relation pairs.
- [Section 1 vs Section 5.5, Table 2] The introduction reports 'up to 157%' improvement in high-negative settings and cites 91.3% versus 80.9% for models trained on 400 relation types. Table 2 contains no 157% value; the largest proportional gain at 99% negatives is 122.5% (14.22 vs 6.39), and the relation counts in Table 2 are 29, 79, 233, 461, and 576, not 400. The 91.36 F1 corresponds to 576 relation types, not 400. Please align the claims with the reported tables and remove or justify the 157% figure.
- [Section 5.2.1 and Table 2] The paper states that bold results are statistically significant (p < 0.05) and underlined results are highest but not significant, yet Table 2 and the surrounding text do not mark significance or report variance despite Section 5.3 saying models were trained and evaluated three times. Without confidence intervals or a significance test, differences between adjacent rows (e.g., 461 vs 576 relation types at 90% negatives: 66.48 vs 66.52) cannot be assessed. Please report per-condition variance and apply the stated significance testing to Table 2.
minor comments (6)
- [Section 1] Typos: 'farther explain' should be 'further explain' and 'summerize' should be 'summarize'.
- [Appendix A, Table 3] The table header 'RECESS-FS' appears to be a typo for 'REBEL-FS'.
- [Section 5.2.2] The Siamese evaluation does not specify how the cosine-similarity threshold is chosen; since F1 depends on the threshold, please state whether it is tuned on a development set and report the threshold or the threshold selection procedure.
- [References] The reference list contains duplicated FewRel 2.0 entries (Gao et al., 2019 and Gao et al., 2020) and several incompletely formatted entries (e.g., the Phi-3 technical report and 'Textbooks Are All You Need'); please clean up the bibliography.
- [Section 5.1 and Table 2] Section 5.1 says relation types are varied as 10, 50, 100, 200, 400, while Table 2 uses 29, 79, 233, 461, 576; this inconsistency should be resolved.
- [Figure 4] The caption for Figure 4 does not explain the dashed lines; the text does, but please make the caption self-contained and note that the reported metric is accuracy.
Circularity Check
No significant circularity; the diversity claim is empirically tested against a fixed held-out test set, and self-citations are not load-bearing.
full rationale
The paper's central claim is that increasing the number of training relation types improves few-shot relation classification generalization at fixed data size. This is tested empirically in Section 5.5 (Table 2) by training on groups of 29, 79, 233, 461, and 576 relation types, all with 100K examples, and measuring F1 on a test set whose positive/negative composition is varied. The test set is the REBEL-FS held-out split of 278 relations, not the training relations, so the reported F1 scores are not definitionally tied to the training input. No fitted parameter is renamed as a prediction; the model is trained and evaluated, and the F1 numbers are measurements. The paper does contain self-citations (e.g., Cohen et al. 2022 for the siamese setup, Sabo et al. 2021a for TACRED-FS), but they support the method and benchmark choices, not the diversity hypothesis itself. The diversity hypothesis is argued from examples and tested with controlled experiments, not derived from a self-citation chain or from a definition that assumes the conclusion. The skeptic's concern that varying relation count changes negative-pair difficulty is a validity threat or confound, not a circularity: it proposes an alternative explanation for the empirical gain, but the result is not equivalent to its input by construction. Therefore no circular step is present, and the paper is self-contained with respect to its main empirical claim.
Assumptions & free parameters
free parameters (2)
- relation frequency threshold for train split =
40 examples
- training positive-to-negative pair ratio =
50% positive, 50% negative
assumptions (4)
- domain assumption BERT and SpanBERT pretrained representations provide a suitable basis for encoding and comparing relation mentions
- domain assumption REBEL silver-standard annotations, though noisy, are accurate enough to learn relation comparison
- domain assumption The REBEL relation inventory provides sufficient coverage of the space of semantic relations for the diversity manipulation to be meaningful
- domain assumption Negative pairs sampled from different relations are a valid proxy for NOTA and negative instances at test time
Cite this review
Pith. "Pith review of Diversity Over Quantity: A Lesson From Few Shot Relation Classification." pith.science (2026). https://pith.science/paper/L6FR6KWK
@misc{pith2026241205434,
author = {Pith},
title = {Pith review of: Diversity Over Quantity: A Lesson From Few Shot Relation Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/L6FR6KWK}},
note = {Machine review of arXiv:2412.05434}
}
read the original abstract
In few-shot relation classification (FSRC), models must generalize to novel relations with only a few labeled examples. While much of the recent progress in NLP has focused on scaling data size, we argue that diversity in relation types is more crucial for FSRC performance. In this work, we demonstrate that training on a diverse set of relations significantly enhances a model's ability to generalize to unseen relations, even when the overall dataset size remains fixed. We introduce REBEL-FS, a new FSRC benchmark that incorporates an order of magnitude more relation types than existing datasets. Through systematic experiments, we show that increasing the diversity of relation types in the training data leads to consistent gains in performance across various few-shot learning scenarios, including high-negative settings. Our findings challenge the common assumption that more data alone leads to better performance and suggest that targeted data curation focused on diversity can substantially reduce the need for large-scale datasets in FSRC.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Livio Baldini Soares , Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. https://doi.org/10.18653/v1/p19-1279 Matching the Blanks: Distributional Similarity for Relation Learning . pages 2895--2905
-
[2]
Philipp Borchert, Jochen De Weerdt , Kristof Coussement, Arno De Caigny , and Marie-Francine Moens. 2023. http://arxiv.org/abs/2310.12024 CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation
arXiv 2023
-
[3]
Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Qianqian Xie, Aidan Gilson, Maxwell Singer, Ron A Adelman, Rui Zhang, Zhiyong Lu, and Hua Xu. 2023. https://arxiv.org/abs/2305.16326v2 Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations
arXiv 2023
-
[4]
Amir Cohen, Hila Gonen, Ori Shapira, Ran Levy, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.259 M c P hra S y: Multi-context phrase similarity and clustering . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3538--3550, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics
-
[5]
Amir DN Cohen, Shachar Rosenman, and Yoav Goldberg. 2020. http://arxiv.org/abs/2010.04829 Relation Classification as Two-way Span-Prediction . In Automated Knowledge Base Construction
work page Pith review arXiv 2020
-
[6]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . ACL
arXiv 2018
-
[8]
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. https://doi.org/10.18653/v1/d19-1649 Fewrel 2.0: Towards more challenging few-shot relation classification . In EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Pr...
-
[9]
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \' e sar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo De, Rosa Olli, Saarikivi Adil, Salim Shital, Shah Harkirat, Singh Behl, Xin Wang, S \' e bastien Bubeck, Ronen Eldan, Adam Tauman, Kalai Yin, Tat Lee, and Yuanzhi Li. 2023. http://arxiv.org/abs/2306.11644 T...
arXiv 2023
Show all 27 references
-
[10]
Muhammad Usman Hadi, Qasem Al Tashi, Rizwan Qureshi, Abbas Shah, Amgad Muneer, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Syed Zohaib Hassan, Maged Shoman, Jia Wu, Seyedali Mirjalili, Mubarak Shah, Qasem Al-Tashi, and Mohammed Ali Al-Garadi. 2024. https:...
2024 doi
-
[12]
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2020. https://doi.org/10.18653/v1/d18-1514 Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation . In Proceedings of the 2018 Conference on Emp...
2020 doi
-
[13]
S \' e aghdha, Sebastian Pad \' o , Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O. S \' e aghdha, Sebastian Pad \' o , Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. http://arxiv.org/abs/1911.10422 SemEval-2010 task 8: Multi-way classification of semantic relations bet...
2010 arXiv
-
[14]
Pere-Llu \' i s Huguet Cabot and Roberto Navigli. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.204 REBEL: Relation Extraction By End-to-end Language generation . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2370--2381
2021 doi
-
[15]
Weld, Luke Zettlemoyer, and Omer Levy
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. https://doi.org/10.1162/tacl_a_00300 SpanBERT: Improving Pre-training by Representing and Predicting Spans . Transactions of the Association for Computational Linguistics, 8:64--77
2020 doi
-
[16]
Xingzuo Li, Kehai Chen, Yunfei Long, and Min Zhang. 2024. http://arxiv.org/abs/2408.13889 Llm with relation classifier for document-level relation extraction
2024 arXiv
-
[17]
Yuanzhi Li, S \' e bastien Bubeck, Ronen Eldan, Allie Del, Giorno Suriya, Gunasekar Yin, and Tat Lee. 2023. http://arxiv.org/abs/2309.05463v1 Textbooks Are All You Need II: phi-1.5
2023 arXiv
-
[18]
Marah Abdin . 2024. http://arxiv.org/abs/2404.14219v3 Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft
2024 arXiv
-
[19]
Yilmazcan Ozyurt, Stefan Feuerriegel, and Ce Zhang. 2024. http://arxiv.org/abs/2310.11085 Document-level in-context few-shot relation extraction via pre-trained language models
2024 arXiv
-
[20]
Nicholas Popovic and Michael F \"a rber. 2022. https://doi.org/10.18653/v1/2022.naacl-main.421 Few-shot document-level relation extraction . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec...
2022 doi
-
[21]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Learning R...
2020 arXiv
-
[22]
Nils Reimers and Iryna Gurevych. 2020. https://doi.org/10.18653/v1/d19-1410 Sentence-BERT: Sentence embeddings using siamese BERT-networks . In EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Nat...
2020 doi
-
[24]
Ofer Sabo, Yanai Elazar, Yoav Goldberg, and Ido Dagan. 2021 b . https://doi.org/10.1162/tacl_a_00392 Revisiting few-shot relation classification: Evaluation data and classification schemes . Transactions of the Association for Computational Linguistics, 9:691--706
2021 doi
-
[25]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. https://doi.org/10.1109/CVPR.2015.7298682 FaceNet: A unified embedding for face recognition and clustering . In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume...
2015
-
[26]
https://github.com/facebookresearch/xformers Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. https://github.com/facebookresearch/xformers ...
-
[27]
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D Manning. 2017. https://doi.org/10.18653/v1/d17-1004 Position-aware attention and supervised data improve slot filling . In EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Procee...
2017 doi
-
[28]
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. http://arxiv.org/abs/2305.11206 LIMA: Less Is More for Alignment
2023 arXiv
-
[29]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.