REVIEW 3 major objections 4 minor 43 references
Neural Snowball for Few-Shot Relation Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Neural Snowball claims that a bootstrapping loop built on a Relational Siamese Network can learn a brand-new relation from as few as five seed sentences, outperforming direct fine-tuning, pure similarity scoring, distant supervision, and…
desk verdict Solid few-shot RE method with realistic evaluation, but the reported F1 gains hinge on corpus/query disjointness the paper never establishes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Relational Siamese Network, a pair encoder $s(x,y)=\sigma(w_s^T (f_s(x)-f_s(y))^2 + b_s)$ trained on existing relations to judge whether two sentences express the same relation; it acts as the precision filter on all candidate instances. The second component is a binary relation classifier $g(x)=\sigma(w^T f(x)+b)$ whose encoder is pre-trained on old relations and whose linear head is fine-tuned on the growing seed set. The snowball alternates Phase 1, which gathers unlabeled sentences sharing entity pairs with the seeds and keeps the top-$K_1$ by RSN score above threshold $\alpha$, with Phase 2, which gathers sentences nominated by the classifier above threshold $\theta$ and keeps the top-$K_2$ by RSN score above threshold $\beta$. The loop repeats, and both CNN and BERT serve as the shared encoder.
What would settle it
Compute exact-string overlap between the released 899,996-instance unlabeled Wikipedia corpus and the FewRel query, validation, and test sentences, remove any overlapping sentences from the unlabeled corpus, and rerun the 5- and 15-seed evaluations; the central claim holds only if the F1 gains over fine-tuning survive this decontamination.
Extended reading notes
Core claim
The central claim is that a new relation can be learned from as few as five labeled instances by iteratively expanding the seed set: first pulling in sentences that share entity pairs with the seeds, then sentences nominated by a freshly fine-tuned binary classifier, and filtering both rounds with a Relational Siamese Network trained on old relations. On FewRel, this yields 72.06 F1 with BERT at 15 seeds versus 68.43 for direct fine-tuning, and 47.26 versus 25.10 at 5 seeds; the method also outperforms pattern-based snowballing, pure RSN similarity, and distant supervision. The paper further reports that RSN achieves 82.15 percent precision at top-5 candidate instances for unseen test relations, and that the iteratively trained classifier keeps higher precision than one trained on the same number of randomly chosen true instances.
Load-bearing premise
The evaluation assumes that the unlabeled Wikipedia corpus used for bootstrapping does not contain the exact sentences that later appear in the query set used to measure F1; if those sets overlap, the snowball can add test sentences to its training data and inflate the reported scores.
Editorial extensions
If this is right
- With as few as five seed instances, a new relation can be added to an extraction system without retraining the encoders; only a new binary head and a scoring pass over unlabeled text are needed.
- The training signal for a new relation comes from unlabeled text plus old labeled relations, not from new human annotations, so the marginal cost of adding a relation is the five seeds.
- The method is most valuable in the extreme few-shot regime: at 5 seeds with BERT it improves F1 from 25.10 to 47.26 over direct fine-tuning, while at 15 seeds the gain is smaller, 68.43 to 72.06.
- Because the snowball loop can run repeatedly on new unlabeled text, the extractor improves over time as long as the RSN keeps selecting high-precision additions; the iteration analysis shows precision stays above a randomly-trained classifier.
- Learned relational similarity filtering supersedes pattern-based snowballing, outperforming the embedding-enhanced BREDS baseline in every seed setting tested.
Reading between the lines
- The paper leaves the RSN frozen after pre-training; a direct extension is to fine-tune or adapt the RSN on the accumulated seed set each iteration, which targets the 'comfort zone' recall limitation the authors report.
- Because each new relation only needs a fresh binary head over a shared encoder, the design composes cleanly with open-ended relation inventories: a knowledge graph can grow by adding heads rather than retraining a global classifier.
- The same entity-pair-recall then similarity-precision loop should transfer to other text domains, such as biomedical abstracts or legal filings, wherever a large labeled relation corpus and an unlabeled entity-tagged corpus are available.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Neural Snowball, a bootstrapping method for few-shot relation extraction. Given a small seed set of instances for a new relation, the method uses a Relational Siamese Network (RSN) pre-trained on existing labeled relations to select high-confidence instances from a large unlabeled Wikipedia corpus, then fine-tunes a binary relation classifier on the accumulated instances, iteratively. The paper evaluates on FewRel-derived splits with CNN and BERT encoders and reports F1 gains over BREDS, fine-tuning, RSN-only, and distant-supervision baselines, along with analyses of RSN selection quality and iteration behavior. The central claim is that Neural Snowball significantly improves few-shot relation learning, with the best reported result being 72.06 F1 versus 68.43 for fine-tuning (BERT, 15 seeds).
Significance. If the quantitative result holds, Neural Snowball is a useful contribution to few-shot relation extraction: the combination of a transfer-learned relational similarity metric with iterative self-training is plausible and addresses a real limitation of fixed-relation supervised models. The paper transparently builds on the authors' prior RSN work, releases code and datasets, and evaluates against a reasonable set of baselines. However, the empirical evidence as reported does not yet support the headline claim because the evaluation protocol leaves open possible train/test overlap, and the main table reports point estimates without variance or significance testing. These issues are fixable, but they are load-bearing for the paper's central quantitative claim.
major comments (3)
- [Datasets and Evaluation Settings; Eqs. (2) and (3)] The evaluation protocol does not establish separation between training and test sentences. First, Sr is sampled from the val/test set and Q is also sampled from the val/test set, but the text never states that the k seed sentences are excluded from Q. Second, the unlabeled corpus T is described as a Wikipedia dump and FewRel instances are also from Wikipedia, but the text never states that T is disjoint from Q. Under Eqs. (2) and (3), Phase 1 and Phase 2 can add exact Q sentences to Sr before the classifier is tested, so the Table 1 F1 gains could be inflated by train/test leakage. The paper must specify that Sr instances are excluded from Q, verify that T contains no Q sentences (or quantify the overlap), and rerun the experiments if overlap is present.
- [Table 1; 'Few-Shot Relation Learning'] The abstract and conclusion claim 'significant improvements,' but Table 1 reports single point estimates of precision, recall, and F1 with no variance, no number of sampled relations or seed draws, and no significance tests. The text says 'we sample one relation' but does not state how many repetitions are averaged. Without these statistics, the reported differences (e.g., 72.06 vs. 68.43 for BERT at 15 seeds) do not support the word 'significant.' Please report means and standard deviations over repeated relation/seed draws and perform significance tests.
- [Pre-training and Fine-tuning; Eq. (6)] In the task definition, the query set Q contains both existing relations and 'other unseen relations' as negatives, but the classifier's negative training batches are drawn only from SN (existing labeled relations). The model therefore never sees a negative example from the unseen-relation class during fine-tuning, and the paper does not report how performance differs on existing-relation versus unseen-relation negatives. Please provide a breakdown of errors by negative type or otherwise justify that negatives from SN are representative of all negatives in Q, since the F1 numbers in Table 1 depend on correct rejection of unseen relations.
minor comments (4)
- [Analysis on Relational Siamese Network; Table 2] The P@N values appear to be based on a single randomly sampled relation; please average over multiple relations and report variance so the reader can assess the stability of the RSN precision estimates.
- [Analysis on Neural Snowball Process; Figure 4] The 'random setting' is described as an upper bound, yet the same paragraph reports that the NS setting maintains higher precision than the random setting; the upper-bound interpretation should be qualified as applying to data diversity or recall, not to overall performance.
- [Experiments] The main results in Table 1 do not state the number of snowball iterations used for Neural Snowball; please specify the iteration count (for example, whether it matches the five iterations shown in Figure 4 or is a tuned value).
- [Datasets and Evaluation Settings] The BREDS baseline is listed in Table 1, but the paper does not describe how BREDS is adapted to the binary few-shot setting; a brief description or citation to an implementation would improve reproducibility.
Circularity Check
No circular derivation: Neural Snowball is an empirical system evaluated on externally labeled FewRel; the self-cited RSN is a transparent component, not an input that forces the reported F1.
full rationale
The paper's central claim is empirical: an iterative bootstrapping procedure that uses a relation-similarity metric learned from existing relations to select unlabeled instances for a new relation, fine-tunes a binary classifier, and evaluates on a query set drawn from FewRel with human labels. No equation defines the output in terms of the input. Phase 1 (Eq. 2) selects candidates by shared entity pairs, Phase 2 (Eq. 3) selects candidates by classifier confidence, and RSN scores are used as filters; these are algorithmic selection rules, not fitted predictions renamed as results. The F1 scores in Table 1 compare against baselines on the same query set, and the labels used for evaluation are external human annotations, so the reported numbers do not reduce to the method's own definitions. The Relational Siamese Network is attributed to Wu et al. (2019), whose author list overlaps with the present paper, but it is used as a modular component rather than as a premise that establishes the few-shot result; moreover, the paper independently evaluates RSN's selection precision in Table 2. The unlabeled corpus T and the query set Q are both Wikipedia-derived, and the paper does not state disjointness, which is a potential data-leakage/correctness concern, not a circularity by construction: even if overlap existed, the evaluation would be contaminated rather than logically equal to the method's inputs. Hyperparameter tuning on the validation set is standard practice and does not constitute fitting the test result. The paper's stated limitations (RSN is fixed during new-relation learning; the snowball stays in a 'comfort zone' and limits recall) are empirical weaknesses, not admissions that the derivation is circular. Overall, the derivation chain is self-contained and the empirical claim stands or falls on experimental validity, not on circular reasoning.
Assumptions & free parameters
free parameters (6)
- RSN thresholds alpha and beta =
0.5
- Classifier threshold theta for Phase 2 candidates =
0.9
- Number of added instances K1 and K2 =
5 each
- Negative sampling coefficient mu =
0.2 (CNN), 0.5 (BERT)
- Fine-tuning epochs, batch size, learning rate =
e=50, bs=10, lr=0.05 (CNN), 0.01 (BERT)
- Final classification thresholds =
0.5 for Fine-tuning and Neural Snowball, 0.7 for RSN baseline
assumptions (3)
- domain assumption The 64 FewRel training relations provide enough relational variety that a siamese network trained on them learns a similarity metric that transfers to unseen relations.
- domain assumption The unlabeled Wikipedia corpus is disjoint from the evaluation query set.
- domain assumption Entity tagging in the unlabeled corpus is correct, so instances sharing an entity pair with seeds are plausible candidates for the same relation.
Cite this review
Pith. "Pith review of Neural Snowball for Few-Shot Relation Learning." pith.science (2026). https://pith.science/paper/G3XKV3X2
@misc{pith2026190811007,
author = {Pith},
title = {Pith review of: Neural Snowball for Few-Shot Relation Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/G3XKV3X2}},
note = {Machine review of arXiv:1908.11007}
}
read the original abstract
Knowledge graphs typically undergo open-ended growth of new relations. This cannot be well handled by relation extraction that focuses on pre-defined relations with sufficient training data. To address new relations with few-shot instances, we propose a novel bootstrapping approach, Neural Snowball, to learn new relations by transferring semantic knowledge about existing relations. More specifically, we use Relational Siamese Networks (RSN) to learn the metric of relational similarities between instances based on existing relations and their labeled data. Afterwards, given a new relation and its few-shot instances, we use RSN to accumulate reliable instances from unlabeled corpora; these instances are used to train a relation classifier, which can further identify new facts of the new relation. The process is conducted iteratively like a snowball. Experiments show that our model can gather high-quality instances for better few-shot relation learning and achieves significant improvement compared to baselines. Codes and datasets are released on https://github.com/thunlp/Neural-Snowball.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Agichtein, E., and Gravano, L. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of JCDL , 85--94
work page 2000
-
[3]
J.; Soderland, S.; Broadhead, M.; and Etzioni, O
Banko, M.; Cafarella, M. J.; Soderland, S.; Broadhead, M.; and Etzioni, O. 2007. Open information extraction from the web. In Proceedings of IJCAI , 2670--2676
work page 2007
-
[4]
Batista, D. S.; Martins, B.; and Silva, M. J. 2015. Semi-supervised bootstrapping of relationship extractors with distributional semantics. In Proceedings of EMNLP , 499--504
work page 2015
-
[5]
Bengio, Y. 2012. Deep learning of representations for unsupervised and transfer learning. In Proceedings of the Workshop on Unsupervised and Transfer Learning of ICML , 17--36
work page 2012
-
[6]
Bollacker, K.; Evans, C.; Paritosh, P.; Sturge, T.; and Taylor, J. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of SIGMOD , 1247--1250
work page 2008
-
[7]
Brin, S. 1998. Extracting patterns and relations from the world wide web. In Proceedings of International Workshop on The World Wide Web and Databases , 172--183
work page 1998
-
[8]
a ckinger, E.; and Shah, R. 1994. Signature verification using a
Bromley, J.; Guyon, I.; LeCun, Y.; S \"a ckinger, E.; and Shah, R. 1994. Signature verification using a" siamese" time delay neural network. In Proceedings of NIPS , 737--744
work page 1994
Show all 43 references
-
[9]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , 4171--4186
2019
-
[10]
ElSahar, H.; Demidova, E.; Gottschalk, S.; Gravier, C.; and Laforest, F. 2017. Unsupervised open relation extraction. In Proceedings of ESWC , 12--16
2017
-
[11]
Finn, C.; Abbeel, P.; and Levine, S. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of ICML , 1126--1135
2017
-
[12]
French, G.; Mackiewicz, M.; and Fisher, M. 2017. Self-ensembling for visual domain adaptation. arXiv preprint arXiv:1706.05208
2017 arXiv
-
[13]
R.; Yu, M.; and Dredze, M
Gormley, M. R.; Yu, M.; and Dredze, M. 2015. Improved relation extraction with feature-rich compositional embedding models. In Proceedings of EMNLP , 1774--1784
2015
-
[14]
Han, X.; Zhu, H.; Yu, P.; Wang, Z.; Yao, Y.; Liu, Z.; and Sun, M. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP , 4803--4809
2018
-
[15]
Hoffmann, R.; Zhang, C.; Ling, X.; Zettlemoyer, L.; and Weld, D. S. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of ACL-HLT , 541--550
2011
-
[16]
P., and Ba, J
Kingma, D. P., and Ba, J. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR
2015
-
[17]
Koch, G.; Zemel, R.; and Salakhutdinov, R. 2015. Siamese neural networks for one-shot image recognition. In Proceedings of the Workshop of ICML
2015
-
[18]
Lin, H.; Yan, J.; Qu, M.; and Ren, X. 2019. Learning dual retrieval module for semi-supervised relation extraction. In Proceedings of WWW , 1073--1083
2019
-
[19]
Liu, Y.; Wei, F.; Li, S.; Ji, H.; Zhou, M.; and Houfeng, W. 2015. A dependency-based neural network for relation classification. In Proceedings of ACL-IJCNLP , 285--290
2015
-
[20]
Miller, G. A. 1995. Wordnet: a lexical database for english. Communications of the ACM 38(11):39--41
1995
-
[21]
Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL-IJCNLP , 1003--1011
2009
-
[22]
Mueller, J., and Thyagarajan, A. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of AAAI
2016
-
[23]
Munkhdalai, T., and Yu, H. 2017. Meta networks. In Proceedings of ICML , 2554--2563
2017
-
[24]
Nakashole, N.; Theobald, M.; and Weikum, G. 2011. Scalable knowledge harvesting with high precision and high recall. In Proceedings of WSDM , 227--236
2011
-
[25]
H., and Grishman, R
Nguyen, T. H., and Grishman, R. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the Workshop on Vector Space Modeling for NLP , 39--48
2015
-
[26]
Pantel, P., and Pennacchiotti, M. 2006. Espresso: Leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of COLING/ACL , 113--120
2006
-
[27]
Qiao, S.; Liu, C.; Shen, W.; and Yuille, A. L. 2018. Few-shot image recognition by predicting parameters from activations. In Proceedings of CVPR , 7229--7238
2018
-
[28]
Ravi, S., and Larochelle, H. 2017. Optimization as a model for few-shot learning. In Proceedings of ICLR
2017
-
[29]
Riedel, S.; Yao, L.; and McCallum, A. 2010. Modeling relations and their mentions without labeled text. In Proceedings of ECML-PKDD , 148--163
2010
-
[30]
Rosenberg, C.; Hebert, M.; and Schneiderman, H. 2005. Semi-supervised self-training of object detection models. In Proceedings of WACV , 29--36
2005
-
[31]
Rozenfeld, B., and Feldman, R. 2008. Self-supervised relation extraction from the web. KAIS 17--33
2008
-
[32]
Shinyama, Y., and Sekine, S. 2006. Preemptive information extraction using unrestricted relation discovery. In Proceedings of NAACL-HLT , 304--311
2006
-
[33]
Snell, J.; Swersky, K.; and Zemel, R. 2017. Prototypical networks for few-shot learning. In Proceedings of NIPS , 4077--4087
2017
-
[34]
D.; and Ng, A
Socher, R.; Huval, B.; Manning, C. D.; and Ng, A. Y. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP-CoNLL , 1201--1211
2012
-
[35]
Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016. Matching networks for one shot learning. In Proceedings of NIPS , 3630--3638
2016
-
[36]
Vrande c i \'c , D., and Kr \"o tzsch, M. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM 57(10):78--85
2014
-
[37]
Wu, R.; Yao, Y.; Han, X.; Xie, R.; Liu, Z.; Lin, F.; Lin, L.; and Sun, M. 2019. Open relation extraction: Relational knowledge transfer from supervised data to unsupervised data. In Proceedings of EMNLP-IJCNLP , 219--228
2019
-
[38]
Xu, Y.; Mou, L.; Li, G.; Chen, Y.; Peng, H.; and Jin, Z. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of EMNLP , 1785--1794
2015
-
[39]
Yao, L.; Haghighi, A.; Riedel, S.; and McCallum, A. 2011. Structured relation discovery using generative models. In Proceedings of EMNLP , 1456--1466
2011
-
[40]
Zelenko, D.; Aone, C.; and Richardella, A. 2003. Kernel methods for relation extraction. JMLR 1083--1106
2003
-
[41]
Zeng, D.; Liu, K.; Lai, S.; Zhou, G.; and Zhao, J. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING , 2335--2344
2014
-
[42]
Zhang, X.; Sung, F.; Qiang, Y.; Yang, Y.; and Hospedales, T. M. 2018. Deep comparison: Relation columns for few-shot learning. arXiv preprint arXiv:1811.07100
2018 arXiv
-
[43]
Zhu, J.; Nie, Z.; Liu, X.; Zhang, B.; and Wen, J.-R. 2009. Statsnowball: a statistical approach to extracting entity relationships. In Proceedings of WWW , 101--110
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.