Pith. sign in

REVIEW 3 major objections 4 minor 43 references

Neural Snowball for Few-Shot Relation Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Neural Snowball claims that a bootstrapping loop built on a Relational Siamese Network can learn a brand-new relation from as few as five seed sentences, outperforming direct fine-tuning, pure similarity scoring, distant supervision, and…

desk verdict Solid few-shot RE method with realistic evaluation, but the reported F1 gains hinge on corpus/query disjointness the paper never establishes. read the letter →

arxiv 1908.11007 v2 pith:G3XKV3X2 submitted 2019-08-29 cs.CL cs.LG

classification cs.CLcs.LG
keywords few-shotrelationextractionbootstrappingRelationalSiameseNetworktransferlearningclassifierFewRelknowledgegraphsnowballmethod
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a knowledge-graph relation never seen during training can be extracted from text using only a handful of seed sentences, by bootstrapping: start from the seeds, pull in sentences from a large unlabeled corpus that look relationally similar, train a binary classifier on the growing set, and repeat. The key trick is to pre-train a relational similarity network on old, well-annotated relations so it can judge similarity even for a relation it has never seen. If the claim holds, knowledge graphs can grow new relations without large new annotation efforts, and the same extraction machinery can be reused for each new relation. On the FewRel benchmark the method beats direct fine-tuning, pure similarity scoring, distant supervision, and a classic pattern-based snowball, with the largest gains in the 5-seed regime.

What carries the argument

The load-bearing mechanism is the Relational Siamese Network, a pair encoder $s(x,y)=\sigma(w_s^T (f_s(x)-f_s(y))^2 + b_s)$ trained on existing relations to judge whether two sentences express the same relation; it acts as the precision filter on all candidate instances. The second component is a binary relation classifier $g(x)=\sigma(w^T f(x)+b)$ whose encoder is pre-trained on old relations and whose linear head is fine-tuned on the growing seed set. The snowball alternates Phase 1, which gathers unlabeled sentences sharing entity pairs with the seeds and keeps the top-$K_1$ by RSN score above threshold $\alpha$, with Phase 2, which gathers sentences nominated by the classifier above threshold $\theta$ and keeps the top-$K_2$ by RSN score above threshold $\beta$. The loop repeats, and both CNN and BERT serve as the shared encoder.

What would settle it

Compute exact-string overlap between the released 899,996-instance unlabeled Wikipedia corpus and the FewRel query, validation, and test sentences, remove any overlapping sentences from the unlabeled corpus, and rerun the 5- and 15-seed evaluations; the central claim holds only if the F1 gains over fine-tuning survive this decontamination.

Watch

Extended reading notes

Core claim

The central claim is that a new relation can be learned from as few as five labeled instances by iteratively expanding the seed set: first pulling in sentences that share entity pairs with the seeds, then sentences nominated by a freshly fine-tuned binary classifier, and filtering both rounds with a Relational Siamese Network trained on old relations. On FewRel, this yields 72.06 F1 with BERT at 15 seeds versus 68.43 for direct fine-tuning, and 47.26 versus 25.10 at 5 seeds; the method also outperforms pattern-based snowballing, pure RSN similarity, and distant supervision. The paper further reports that RSN achieves 82.15 percent precision at top-5 candidate instances for unseen test relations, and that the iteratively trained classifier keeps higher precision than one trained on the same number of randomly chosen true instances.

Load-bearing premise

The evaluation assumes that the unlabeled Wikipedia corpus used for bootstrapping does not contain the exact sentences that later appear in the query set used to measure F1; if those sets overlap, the snowball can add test sentences to its training data and inflate the reported scores.

Editorial extensions

If this is right

  • With as few as five seed instances, a new relation can be added to an extraction system without retraining the encoders; only a new binary head and a scoring pass over unlabeled text are needed.
  • The training signal for a new relation comes from unlabeled text plus old labeled relations, not from new human annotations, so the marginal cost of adding a relation is the five seeds.
  • The method is most valuable in the extreme few-shot regime: at 5 seeds with BERT it improves F1 from 25.10 to 47.26 over direct fine-tuning, while at 15 seeds the gain is smaller, 68.43 to 72.06.
  • Because the snowball loop can run repeatedly on new unlabeled text, the extractor improves over time as long as the RSN keeps selecting high-precision additions; the iteration analysis shows precision stays above a randomly-trained classifier.
  • Learned relational similarity filtering supersedes pattern-based snowballing, outperforming the embedding-enhanced BREDS baseline in every seed setting tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the RSN frozen after pre-training; a direct extension is to fine-tune or adapt the RSN on the accumulated seed set each iteration, which targets the 'comfort zone' recall limitation the authors report.
  • Because each new relation only needs a fresh binary head over a shared encoder, the design composes cleanly with open-ended relation inventories: a knowledge graph can grow by adding heads rather than retraining a global classifier.
  • The same entity-pair-recall then similarity-precision loop should transfer to other text domains, such as biomedical abstracts or legal filings, wherever a large labeled relation corpus and an unlabeled entity-tagged corpus are available.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Neural Snowball, a bootstrapping method for few-shot relation extraction. Given a small seed set of instances for a new relation, the method uses a Relational Siamese Network (RSN) pre-trained on existing labeled relations to select high-confidence instances from a large unlabeled Wikipedia corpus, then fine-tunes a binary relation classifier on the accumulated instances, iteratively. The paper evaluates on FewRel-derived splits with CNN and BERT encoders and reports F1 gains over BREDS, fine-tuning, RSN-only, and distant-supervision baselines, along with analyses of RSN selection quality and iteration behavior. The central claim is that Neural Snowball significantly improves few-shot relation learning, with the best reported result being 72.06 F1 versus 68.43 for fine-tuning (BERT, 15 seeds).

Significance. If the quantitative result holds, Neural Snowball is a useful contribution to few-shot relation extraction: the combination of a transfer-learned relational similarity metric with iterative self-training is plausible and addresses a real limitation of fixed-relation supervised models. The paper transparently builds on the authors' prior RSN work, releases code and datasets, and evaluates against a reasonable set of baselines. However, the empirical evidence as reported does not yet support the headline claim because the evaluation protocol leaves open possible train/test overlap, and the main table reports point estimates without variance or significance testing. These issues are fixable, but they are load-bearing for the paper's central quantitative claim.

major comments (3)
  1. [Datasets and Evaluation Settings; Eqs. (2) and (3)] The evaluation protocol does not establish separation between training and test sentences. First, Sr is sampled from the val/test set and Q is also sampled from the val/test set, but the text never states that the k seed sentences are excluded from Q. Second, the unlabeled corpus T is described as a Wikipedia dump and FewRel instances are also from Wikipedia, but the text never states that T is disjoint from Q. Under Eqs. (2) and (3), Phase 1 and Phase 2 can add exact Q sentences to Sr before the classifier is tested, so the Table 1 F1 gains could be inflated by train/test leakage. The paper must specify that Sr instances are excluded from Q, verify that T contains no Q sentences (or quantify the overlap), and rerun the experiments if overlap is present.
  2. [Table 1; 'Few-Shot Relation Learning'] The abstract and conclusion claim 'significant improvements,' but Table 1 reports single point estimates of precision, recall, and F1 with no variance, no number of sampled relations or seed draws, and no significance tests. The text says 'we sample one relation' but does not state how many repetitions are averaged. Without these statistics, the reported differences (e.g., 72.06 vs. 68.43 for BERT at 15 seeds) do not support the word 'significant.' Please report means and standard deviations over repeated relation/seed draws and perform significance tests.
  3. [Pre-training and Fine-tuning; Eq. (6)] In the task definition, the query set Q contains both existing relations and 'other unseen relations' as negatives, but the classifier's negative training batches are drawn only from SN (existing labeled relations). The model therefore never sees a negative example from the unseen-relation class during fine-tuning, and the paper does not report how performance differs on existing-relation versus unseen-relation negatives. Please provide a breakdown of errors by negative type or otherwise justify that negatives from SN are representative of all negatives in Q, since the F1 numbers in Table 1 depend on correct rejection of unseen relations.
minor comments (4)
  1. [Analysis on Relational Siamese Network; Table 2] The P@N values appear to be based on a single randomly sampled relation; please average over multiple relations and report variance so the reader can assess the stability of the RSN precision estimates.
  2. [Analysis on Neural Snowball Process; Figure 4] The 'random setting' is described as an upper bound, yet the same paragraph reports that the NS setting maintains higher precision than the random setting; the upper-bound interpretation should be qualified as applying to data diversity or recall, not to overall performance.
  3. [Experiments] The main results in Table 1 do not state the number of snowball iterations used for Neural Snowball; please specify the iteration count (for example, whether it matches the five iterations shown in Figure 4 or is a tuned value).
  4. [Datasets and Evaluation Settings] The BREDS baseline is listed in Table 1, but the paper does not describe how BREDS is adapted to the binary few-shot setting; a brief description or citation to an implementation would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: Neural Snowball is an empirical system evaluated on externally labeled FewRel; the self-cited RSN is a transparent component, not an input that forces the reported F1.

full rationale

The paper's central claim is empirical: an iterative bootstrapping procedure that uses a relation-similarity metric learned from existing relations to select unlabeled instances for a new relation, fine-tunes a binary classifier, and evaluates on a query set drawn from FewRel with human labels. No equation defines the output in terms of the input. Phase 1 (Eq. 2) selects candidates by shared entity pairs, Phase 2 (Eq. 3) selects candidates by classifier confidence, and RSN scores are used as filters; these are algorithmic selection rules, not fitted predictions renamed as results. The F1 scores in Table 1 compare against baselines on the same query set, and the labels used for evaluation are external human annotations, so the reported numbers do not reduce to the method's own definitions. The Relational Siamese Network is attributed to Wu et al. (2019), whose author list overlaps with the present paper, but it is used as a modular component rather than as a premise that establishes the few-shot result; moreover, the paper independently evaluates RSN's selection precision in Table 2. The unlabeled corpus T and the query set Q are both Wikipedia-derived, and the paper does not state disjointness, which is a potential data-leakage/correctness concern, not a circularity by construction: even if overlap existed, the evaluation would be contaminated rather than logically equal to the method's inputs. Hyperparameter tuning on the validation set is standard practice and does not constitute fitting the test result. The paper's stated limitations (RSN is fixed during new-relation learning; the snowball stays in a 'comfort zone' and limits recall) are empirical weaknesses, not admissions that the derivation is circular. Overall, the derivation chain is self-contained and the empirical claim stands or falls on experimental validity, not on circular reasoning.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The reported F1 depends on hand-set thresholds (alpha, beta, theta), growth caps (K1, K2), and fine-tuning hyperparameters, all grid-searched on validation, plus two domain assumptions: that RSN's relational metric transfers from 64 training relations to unseen ones, and that the unlabeled corpus is cleanly separated from the query set. No new physical or conceptual entities are introduced.

free parameters (6)
  • RSN thresholds alpha and beta = 0.5
    Grid-searched on the validation set; instances below these scores are discarded, so the values directly determine which unlabeled sentences become training data and thus the final F1.
  • Classifier threshold theta for Phase 2 candidates = 0.9
    Grid-searched on validation; sentences with g(x) below 0.9 are never proposed to RSN, controlling coverage versus precision.
  • Number of added instances K1 and K2 = 5 each
    Grid-searched on validation; caps how fast the seed set grows each iteration.
  • Negative sampling coefficient mu = 0.2 (CNN), 0.5 (BERT)
    Grid-searched; controls the weight of negative examples when fine-tuning the classifier, affecting the precision-recall tradeoff.
  • Fine-tuning epochs, batch size, learning rate = e=50, bs=10, lr=0.05 (CNN), 0.01 (BERT)
    Grid-searched; standard training hyperparameters for the binary classifier.
  • Final classification thresholds = 0.5 for Fine-tuning and Neural Snowball, 0.7 for RSN baseline
    Chosen for evaluation; changes the precision-recall operating point and makes comparisons across methods not fully controlled.
assumptions (3)
  • domain assumption The 64 FewRel training relations provide enough relational variety that a siamese network trained on them learns a similarity metric that transfers to unseen relations.
    The whole transfer mechanism assumes generic relational similarity learned on historical relations is meaningful for new relations; invoked in pre-training of RSN in 'Neural Modules'.
  • domain assumption The unlabeled Wikipedia corpus is disjoint from the evaluation query set.
    Load-bearing for clean evaluation; the paper never states this disjointness, and both T and Q derive from Wikipedia (see 'Datasets and Evaluation Settings').
  • domain assumption Entity tagging in the unlabeled corpus is correct, so instances sharing an entity pair with seeds are plausible candidates for the same relation.
    Phase 1 builds candidate set C1 from entity-pair matches; wrong tags would inject noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Snowball for Few-Shot Relation Learning." pith.science (2026). https://pith.science/paper/G3XKV3X2

@misc{pith2026190811007,
  author       = {Pith},
  title        = {Pith review of: Neural Snowball for Few-Shot Relation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3XKV3X2}},
  note         = {Machine review of arXiv:1908.11007}
}
read the original abstract

Knowledge graphs typically undergo open-ended growth of new relations. This cannot be well handled by relation extraction that focuses on pre-defined relations with sufficient training data. To address new relations with few-shot instances, we propose a novel bootstrapping approach, Neural Snowball, to learn new relations by transferring semantic knowledge about existing relations. More specifically, we use Relational Siamese Networks (RSN) to learn the metric of relational similarities between instances based on existing relations and their labeled data. Afterwards, given a new relation and its few-shot instances, we use RSN to accumulate reliable instances from unlabeled corpora; these instances are used to train a relation classifier, which can further identify new facts of the new relation. The process is conducted iteratively like a snowball. Experiments show that our model can gather high-quality instances for better few-shot relation learning and achieves significant improvement compared to baselines. Codes and datasets are released on https://github.com/thunlp/Neural-Snowball.

Figures

Figures reproduced from arXiv: 1908.11007 by the authors.

Figure 1
Figure 1. An illustration of how Neural Snowball utilizes [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of Neural Snowball with examples of the relation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The architecture of Relational Siamese Network [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Evaluation results on each iteration of Neural [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 41 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Agichtein, E., and Gravano, L. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of JCDL , 85--94

  3. [3]

    J.; Soderland, S.; Broadhead, M.; and Etzioni, O

    Banko, M.; Cafarella, M. J.; Soderland, S.; Broadhead, M.; and Etzioni, O. 2007. Open information extraction from the web. In Proceedings of IJCAI , 2670--2676

  4. [4]

    S.; Martins, B.; and Silva, M

    Batista, D. S.; Martins, B.; and Silva, M. J. 2015. Semi-supervised bootstrapping of relationship extractors with distributional semantics. In Proceedings of EMNLP , 499--504

  5. [5]

    Bengio, Y. 2012. Deep learning of representations for unsupervised and transfer learning. In Proceedings of the Workshop on Unsupervised and Transfer Learning of ICML , 17--36

  6. [6]

    Bollacker, K.; Evans, C.; Paritosh, P.; Sturge, T.; and Taylor, J. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of SIGMOD , 1247--1250

  7. [7]

    Brin, S. 1998. Extracting patterns and relations from the world wide web. In Proceedings of International Workshop on The World Wide Web and Databases , 172--183

  8. [8]

    a ckinger, E.; and Shah, R. 1994. Signature verification using a

    Bromley, J.; Guyon, I.; LeCun, Y.; S \"a ckinger, E.; and Shah, R. 1994. Signature verification using a" siamese" time delay neural network. In Proceedings of NIPS , 737--744

Show all 43 references
  1. [9]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , 4171--4186

  2. [10]

    ElSahar, H.; Demidova, E.; Gottschalk, S.; Gravier, C.; and Laforest, F. 2017. Unsupervised open relation extraction. In Proceedings of ESWC , 12--16

  3. [11]

    Finn, C.; Abbeel, P.; and Levine, S. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of ICML , 1126--1135

  4. [12]

    French, G.; Mackiewicz, M.; and Fisher, M. 2017. Self-ensembling for visual domain adaptation. arXiv preprint arXiv:1706.05208

  5. [13]

    R.; Yu, M.; and Dredze, M

    Gormley, M. R.; Yu, M.; and Dredze, M. 2015. Improved relation extraction with feature-rich compositional embedding models. In Proceedings of EMNLP , 1774--1784

  6. [14]

    Han, X.; Zhu, H.; Yu, P.; Wang, Z.; Yao, Y.; Liu, Z.; and Sun, M. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of EMNLP , 4803--4809

  7. [15]

    Hoffmann, R.; Zhang, C.; Ling, X.; Zettlemoyer, L.; and Weld, D. S. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of ACL-HLT , 541--550

  8. [16]

    P., and Ba, J

    Kingma, D. P., and Ba, J. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR

  9. [17]

    Koch, G.; Zemel, R.; and Salakhutdinov, R. 2015. Siamese neural networks for one-shot image recognition. In Proceedings of the Workshop of ICML

  10. [18]

    Lin, H.; Yan, J.; Qu, M.; and Ren, X. 2019. Learning dual retrieval module for semi-supervised relation extraction. In Proceedings of WWW , 1073--1083

  11. [19]

    Liu, Y.; Wei, F.; Li, S.; Ji, H.; Zhou, M.; and Houfeng, W. 2015. A dependency-based neural network for relation classification. In Proceedings of ACL-IJCNLP , 285--290

  12. [20]

    Miller, G. A. 1995. Wordnet: a lexical database for english. Communications of the ACM 38(11):39--41

  13. [21]

    Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL-IJCNLP , 1003--1011

  14. [22]

    Mueller, J., and Thyagarajan, A. 2016. Siamese recurrent architectures for learning sentence similarity. In Proceedings of AAAI

  15. [23]

    Munkhdalai, T., and Yu, H. 2017. Meta networks. In Proceedings of ICML , 2554--2563

  16. [24]

    Nakashole, N.; Theobald, M.; and Weikum, G. 2011. Scalable knowledge harvesting with high precision and high recall. In Proceedings of WSDM , 227--236

  17. [25]

    H., and Grishman, R

    Nguyen, T. H., and Grishman, R. 2015. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the Workshop on Vector Space Modeling for NLP , 39--48

  18. [26]

    Pantel, P., and Pennacchiotti, M. 2006. Espresso: Leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of COLING/ACL , 113--120

  19. [27]

    Qiao, S.; Liu, C.; Shen, W.; and Yuille, A. L. 2018. Few-shot image recognition by predicting parameters from activations. In Proceedings of CVPR , 7229--7238

  20. [28]

    Ravi, S., and Larochelle, H. 2017. Optimization as a model for few-shot learning. In Proceedings of ICLR

  21. [29]

    Riedel, S.; Yao, L.; and McCallum, A. 2010. Modeling relations and their mentions without labeled text. In Proceedings of ECML-PKDD , 148--163

  22. [30]

    Rosenberg, C.; Hebert, M.; and Schneiderman, H. 2005. Semi-supervised self-training of object detection models. In Proceedings of WACV , 29--36

  23. [31]

    Rozenfeld, B., and Feldman, R. 2008. Self-supervised relation extraction from the web. KAIS 17--33

  24. [32]

    Shinyama, Y., and Sekine, S. 2006. Preemptive information extraction using unrestricted relation discovery. In Proceedings of NAACL-HLT , 304--311

  25. [33]

    Snell, J.; Swersky, K.; and Zemel, R. 2017. Prototypical networks for few-shot learning. In Proceedings of NIPS , 4077--4087

  26. [34]

    D.; and Ng, A

    Socher, R.; Huval, B.; Manning, C. D.; and Ng, A. Y. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP-CoNLL , 1201--1211

  27. [35]

    Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D.; et al. 2016. Matching networks for one shot learning. In Proceedings of NIPS , 3630--3638

  28. [36]

    Vrande c i \'c , D., and Kr \"o tzsch, M. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM 57(10):78--85

  29. [37]

    Wu, R.; Yao, Y.; Han, X.; Xie, R.; Liu, Z.; Lin, F.; Lin, L.; and Sun, M. 2019. Open relation extraction: Relational knowledge transfer from supervised data to unsupervised data. In Proceedings of EMNLP-IJCNLP , 219--228

  30. [38]

    Xu, Y.; Mou, L.; Li, G.; Chen, Y.; Peng, H.; and Jin, Z. 2015. Classifying relations via long short term memory networks along shortest dependency paths. In Proceedings of EMNLP , 1785--1794

  31. [39]

    Yao, L.; Haghighi, A.; Riedel, S.; and McCallum, A. 2011. Structured relation discovery using generative models. In Proceedings of EMNLP , 1456--1466

  32. [40]

    Zelenko, D.; Aone, C.; and Richardella, A. 2003. Kernel methods for relation extraction. JMLR 1083--1106

  33. [41]

    Zeng, D.; Liu, K.; Lai, S.; Zhou, G.; and Zhao, J. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING , 2335--2344

  34. [42]

    Zhang, X.; Sung, F.; Qiang, Y.; Yang, Y.; and Hospedales, T. M. 2018. Deep comparison: Relation columns for few-shot learning. arXiv preprint arXiv:1811.07100

  35. [43]

    Zhu, J.; Nie, Z.; Liu, X.; Zhang, B.; and Wen, J.-R. 2009. Statsnowball: a statistical approach to extracting entity relationships. In Proceedings of WWW , 101--110

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.