Pith. sign in

REVIEW 2 major objections 5 minor 37 references

Few-shot Text Classification with Distributional Signatures

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Words may not transfer, but their distributional behavior does.

desk verdict Solid empirical contribution with a real distributional-signature idea, but the robustness theorem is for a different estimator than the one deployed. read the letter →

arxiv 1908.06039 v3 pith:DRPUJCGZ submitted 2019-08-16 cs.CL cs.LG

Yujia Bao , Menghua Wu , Shiyu Chang , Regina Barzilay This is my paper · ORCID
classification cs.CLcs.LG
keywords few-shottextclassificationmeta-learningdistributionalsignaturesattentiongenerationunigramstatisticsridgeregressionrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the transferable knowledge in few-shot text classification is not lexical—which words appear—but distributional—how words are distributed. It claims that word importance for an unseen class can be predicted from unigram statistics such as overall frequency and how skewed a word is across classes, and that a meta-learned generator can turn these noisy statistics into attention weights. Across six datasets, the model outperforms prototypical networks built on lexical knowledge by 20.0% on average in 1-shot classification. If true, this means few-shot learning in text can work even when seen and unseen classes share almost no vocabulary.

What carries the argument

The load-bearing object is the distributional signature, the pair $s(w)=\epsilon/(\epsilon+P(w))$ and $t(w)=H(P(y|w))^{-1}$, where $P(w)$ is estimated over a large source pool and $P(y|w)$ is estimated over the small support set. These two scalars make word importance a function of the word distribution rather than of the word identity: rare words and words that separate classes are upweighted. An attention generator—a biLSTM followed by dot-product attention—fuses the signatures into attention scores that weight pretrained frozen word embeddings; a ridge regressor then learns the episode from the support set through a differentiable closed-form solution. The paper proves that the attention generator's output is invariant to bijective word substitutions that preserve unigram probabilities, which is what makes signature-based attention transferable.

What would settle it

Construct a meta-test split in which no content word appears in both the training and test classes; if 1-shot accuracy falls to lexical-baseline levels or to chance, the claim that unigram distributional signatures transfer would be refuted. Alternatively, compare generated attention against an oracle trained with all labels: if the cosine similarity is no better than the raw statistics alone, the meta-learned mapping is not doing the claimed work.

Watch

Extended reading notes

Core claim

The central discovery is that meta-knowledge built on distributional signatures generalizes across disjoint class sets. The model computes two unigram statistics per word—smoothed inverse frequency over a source pool and inverse entropy of the class distribution given the word over the support set—fuses them with a biLSTM, and turns them into per-word attention. The attention-weighted embedding is fed to a ridge regressor whose closed-form solution is trained from scratch each episode, with the query loss supervising the attention generator. On five text classification datasets and one relation classification dataset, the model reaches 60.1% average 5-way 1-shot accuracy and 78.0% in 5-shot, versus 40.1% and 48.7% for the strongest lexical prototypical-network baseline; ablations show removing either statistic hurts, and adding word embeddings to the attention generator hurts.

Load-bearing premise

The model assumes that word importance for a brand-new class can be predicted from frequency and class-skew statistics, even when the specific informative words have never appeared in the training classes.

Editorial extensions

If this is right

  • Few-shot text classifiers can be deployed on new class sets with very different vocabularies: the reported hard splits include test datasets where roughly 60% of Amazon test tokens never appear in training.
  • The transfer mechanism is portable across downstream learners: augmenting prototypical networks and induction networks with distributional-signature attention improves them as well, not just the ridge regressor.
  • The same meta-trained attention generator transfers across domains, from news topics and product categories to Reuters articles and relation classification.
  • Because the invariance proof holds for any function of unigram counts, other unigram statistics besides the two chosen should inherit the same perturbation robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the negative result when word embeddings are added to the attention generator suggests the transferable signal is an abstraction over statistics, not a feature combination, and that freezing pretrained embedding geometry may be important for generalization.
  • Editorial inference: the frequency–skew decomposition resembles TF-IDF, but the paper learns the combination rather than fixing it; a natural extension is to learn other task-agnostic statistics, such as document length or syntactic cues, in the same framework.
  • Editorial inference: a harder test than the paper's hard splits would be a deliberately constructed split with no content-word overlap, since the hard splits reduce overlap but do not eliminate it entirely.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes a meta-learning approach for few-shot text classification in which attention weights are generated from distributional signatures rather than from lexical content. The attention generator combines the Arora et al. smoothing statistic s(w)=epsilon/(epsilon+P(w)) with a class-skewness statistic t(w)=1/H(P(y|w)), where P(y|w) is estimated over the support set with a regularized linear classifier, and passes the concatenated signatures through a biLSTM followed by dot-product attention. The resulting attention weights are used to form weighted averages of pretrained word embeddings, which are then fed to a ridge regressor trained from scratch per episode; the query loss provides end-to-end supervision for the attention generator. The paper states a theoretical robustness guarantee, namely that the attention generator is invariant to bijective word substitutions that preserve unigram probabilities. It evaluates the method on five text-classification datasets plus FewRel, under hard and easy class splits, and reports consistent, large gains over lexical baselines (e.g., 60.1 vs 40.1 average 5-way 1-shot accuracy vs CNN+PROTO), with ablations, standard deviations, BERT experiments, additional classifiers, and released code.

Significance. If the empirical results hold, the paper makes an important contribution: it demonstrates that meta-knowledge for few-shot NLP can be transferred through distributional statistics rather than through lexical representations, and it provides a practical recipe that improves on prototypical networks by a large margin on six benchmarks. The evaluation is a genuine strength: hard class splits create substantial lexical mismatch between meta-train and meta-test, results are averaged over five seeds with standard deviations, ablations isolate the contribution of each signature, the code is publicly available, and the method is combined with alternative downstream classifiers and with BERT. The theoretical robustness claim, however, is not supported for the model that is actually implemented: Theorem 1 is proved for a count-based MLE estimator of P(y|w), while the deployed t(w) in Eq. (2) uses an embedding-based regularized linear classifier. This mismatch affects the conceptual contribution of the paper, but the empirical comparisons themselves do not depend on Theorem 1, so the appropriate remedy is a major revision rather than rejection.

major comments (2)
  1. [Section 4.3, Eq. (2), Appendix A.1/A.2] Theorem 1 does not apply to the deployed attention generator. The proof in Appendix A.2 establishes invariance of t by showing that the count-based MLE estimate Phat(y|w,S) is unchanged under a unigram-preserving bijection sigma. The implemented t in Eq. (2), however, is H(P(y|w))^{-1} with P(y|w)=softmax(Wpsi(w)), where W is a regularized linear classifier fit on the support set using pretrained word embeddings. Under sigma, the support set is renamed but the embeddings psi(sigma(w)) are generally different from psi(w), and nothing forces the re-fit matrix Wtilde to satisfy softmax(Wpsi(w)) = softmax(Wtilde psi(sigma(w))). Thus the theorem is a property of a stylized model, not of the model evaluated in Section 6. The sentence in Section 4.3 that 'the theorem holds when the input to the attention generator is any function of unigram counts' is also inaccurate for the actual t, which depends on pretrained embeddings and on a fitted classifier. Please either replace t with a count-based estimator and verify that the empirical gains survive, or explicitly scope Theorem 1 to an idealized attention generator and remove the robustness claim attached to the deployed model.
  2. [Section 4.3 and Section A.4] The perturbation class considered in Theorem 1 is a bijective renaming of the vocabulary that preserves unigram probabilities exactly. This is essentially a label-preserving re-labeling of word identities, and it does not model the hard-split regime of Section A.4, where meta-test words are largely absent from the meta-train vocabulary and no such bijection exists on the observed vocabulary. The paper should clarify that Theorem 1 is a consistency property under a restricted re-labeling, not a certificate for the cross-class lexical generalization measured in Table 1. Without this clarification, the phrase 'certified robustness' in Section 4.3 overstates what the theorem guarantees.
minor comments (5)
  1. [Appendix A.6] In the paragraph on lexical mismatch, '2481 of 5249 (47.2%) meta-test tokens are not in meta-test' should read 'not in meta-train.'
  2. [Appendix A.11] In the MAML implementation paragraph, 'Adam with learning rate 103' should read '10^{-3}'; the main text already specifies a learning rate of 0.001.
  3. [Section 6, ablation paragraph] The statement that performance 'drops consistently' for OUR w EBD is stronger than the numbers support: on Reuters the 1-shot difference is 0.1 and the 5-shot difference is 0.2, both within one standard deviation in Tables 8 and 9. Please report this nuance or soften the wording.
  4. [Appendix A.1] The notation psi is overloaded: psi(x) is the average embedding of a sentence, while psi(x_i) is used for the embedding of a single word. Define psi_sentence and psi_word, or use febd for word vectors, to avoid ambiguity in Eq. (2) and its appendix.
  5. [Section 3, source pool] At meta-test time the method uses the entire Ytrain corpus to estimate P(w), while most baselines do not receive an analogous episode-level source pool. A sentence clarifying that only unigram counts are used from this pool, and explaining why this is not the source of the reported gains, would strengthen the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attention generator and ridge regressor are trained and evaluated on external benchmarks, and no predicted quantity is equivalent to a fitted input by construction.

full rationale

The paper's derivation chain is self-contained against external data. The attention generator is trained end-to-end on source-class episodes, and its output is evaluated on held-out test classes from six standard benchmarks; no parameter fitted to a subset is later renamed as a prediction of that subset. The two distributional statistics s(x) and t(x) are hand-specified features, not fitted constants, and the downstream ridge regressor is trained from scratch per episode. The theoretical invariance result (Theorem 1, Section 4.3) is a separate claim about an idealized count-based estimator, and its proof does not feed back into the empirical accuracies; even if the proof's assumptions mismatch the implemented regularized linear classifier in Eq. 2, that is a theory-practice gap rather than a circular reduction. Self-citations in Related Work (e.g., Bao et al. 2018; Guo et al. 2018) are contextual and non-load-bearing. No load-bearing result is justified solely by a self-citation, and the central empirical claim is not equivalent by construction to any input of the method.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method rests on two domain assumptions: unigram statistics are informative about word importance, and these statistics transfer across class sets. The source pool and support set are used to estimate them. The theoretical invariance proof assumes an MLE estimator that does not match the implemented regularized linear classifier, so the proof is an ad hoc assumption for the claim. No new entities are introduced.

free parameters (4)
  • epsilon (smoothing constant in s(x)) = 1e-3
    Hand-chosen constant in Eq. (1), taken from Arora et al. (2016); controls the down-weighting of frequent words.
  • lambda (ridge regularization) = learned during meta-training (log-space), values not reported
    Controls conditioning of the ridge regression closed-form solution in Eq. (6).
  • softmax calibration a, b = learned during meta-training, values not reported
    Temperature and bias for converting regression outputs to probabilities in Eq. (7).
  • support-set classifier stopping criterion = gradient norm < 0.1
    Hyperparameter in the regularized linear classifier (Appendix A.1) used to estimate t(x); chosen by hand.
assumptions (5)
  • domain assumption The unigram statistic s(x) = epsilon/(epsilon + P(w)) is a valid indicator of general word importance (low frequency implies informative).
    Adopted from Arora et al. (2016); the paper uses it to down-weight frequent words, assuming this holds across all target tasks.
  • domain assumption The inverse class-entropy statistic t(x) = 1/H(P(y|x)) estimates class-specific word importance from the support set.
    The paper assumes that words with skewed class distributions in the support set are discriminative in the query set (Section 4.1).
  • domain assumption Distributional signatures transfer across disjoint class sets, even when lexical overlap is low.
    Core assumption of the method; the hard-split experiments (Appendix A.4) create large vocabulary mismatch, and the method's success depends on this transfer.
  • domain assumption The source pool provides a reliable estimate of the unigram distribution P(w) for computing s(x).
    The method uses all training examples as the source pool at meta-test time; if this corpus is unrepresentative, s(x) is unreliable.
  • ad hoc to paper In the proof of Theorem 1, P(y|x) is estimated by maximum likelihood over the support set, yielding invariance under unigram-preserving bijections.
    The implemented t(x) uses a regularized linear classifier (Appendix A.1), not MLE; the paper does not show the classifier estimate is invariant, so the theorem's assumptions do not match the implementation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Few-shot Text Classification with Distributional Signatures." pith.science (2026). https://pith.science/paper/DRPUJCGZ

@misc{pith2026190806039,
  author       = {Pith},
  title        = {Pith review of: Few-shot Text Classification with Distributional Signatures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRPUJCGZ}},
  note         = {Machine review of arXiv:1908.06039}
}
read the original abstract

In this paper, we explore meta-learning for few-shot text classification. Meta-learning has shown strong performance in computer vision, where low-level patterns are transferable across learning tasks. However, directly applying this approach to text is challenging--lexical features highly informative for one task may be insignificant for another. Thus, rather than learning solely from words, our model also leverages their distributional signatures, which encode pertinent word occurrence patterns. Our model is trained within a meta-learning framework to map these signatures into attention scores, which are then used to weight the lexical representations of words. We demonstrate that our model consistently outperforms prototypical networks learned on lexical knowledge (Snell et al., 2017) in both few-shot text classification and relation classification by a significant margin across six benchmark datasets (20.0% on average in 1-shot classification).

Figures

Figures reproduced from arXiv: 1908.06039 by the authors.

Figure 2
Figure 2. Visualization of word importance on example from class fifty in HuffPost headlines. Top: fifty is seen during meta-training; prototyp￾ical network (Snell et al., 2017) finds important words. Middle: fifty is unavailable during meta￾training; it fails to generalize. Bottom: Our model identifies key words for unseen classes. signatures is tf-idf weighting, which explicitly specifies word importance in terms of its fre… view at source ↗
Figure 3
Figure 3. Episode generation. a) Meta-training: First, sample [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Single episode with N = 3, K = 1, L = 2. Rectangles denote input examples. The text inside corresponds to their labels. An episode contains a support set, a query set, and a source pool. first sample N new classes from Y test. Then we sample the support set and the query set from the N classes. We evaluate the average performance on the query set across all testing episodes. Our extension We observe that even though… view at source ↗
Figures from the paper (11 more)
Figure 5
Figure 5. Figure 5: Illustration of our model for an episode with [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Learning curve of CNN+PROTO (left) v.s. OUR (right) on the Reuters dataset. We plot average 5-way 1-shot accuracy over 50 episodes sampled from seen classes (blue) and unseen classes (red). While OUR has weaker representational power, it generalizes better to unseen cl…
Figure 7
Figure 7. Figure 7: PCA visualization of the input representation for the query set of a testing episode ( [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Attention weights generated by our model are specific to task. We visualize our model’s [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Learning curves of INDUCTION NET on Huffpost and Reuters. We plot average 5-way 5-shot accuracy over 50 episodes sampled from seen classes (blue) and unseen classes (red). • Induction network encodes input examples using a biLSTM with self-attentive pool￾ing (Lin et al…
Figure 10
Figure 10. Figure 10: Perplexity during BERT language model finetuning in [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: PCA visualization of the input representation for a testing episode in 20 Newsgroups [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: PCA visualization of the input representation for a testing episode in HuffPost Headlines [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Average cosine similarity to the oracle word importance over the query set of a testing episode with N = 5, K = 5, L = 500. This oracle is estimated using all labeled examples from the N classes. Since examples in HuffPost Headlines are 30 times shorter, the cosine si…
Figure 14
Figure 14. Figure 14: Visualization of the attention generated by our model on 10 query examples from a [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Learning curve for fully-supervised CNN classifier vs [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages

  1. [1]

    How to train your maml

    Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. arXiv preprint arXiv:1810.09502,

    arXiv

  2. [2]

    sci” and “rec,

    hard split : we select train/val/test based on the class hierarchy such that train 7less than 1 second on a single GeForce GTX TITAN X 14 Published as a conference paper at ICLR 2020 Algorithm 1 Meta-training procedure.Ntrain = ⏐⏐Ytrain⏐⏐ is the number of training classes.N <Ntrain is the number of classes of each few-shot task. K,L are the number of supp...

    work page 2017

  3. [5]

    Induction Networks for Few-Shot Text Classification

    Ruiying Geng, Binhua Li, Yongbin Li, Yuxiao Ye, Ping Jian, and Jian Sun. Few-shot text classifica- tion with induction network. arXiv preprint arXiv:1902.10482,

    work page Pith review arXiv 1902

  4. [8]

    Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation

    Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4803–4809,

    work page 2018

  5. [10]

    Attentive task-agnostic meta-learning for few-shot text classification,

    11 Published as a conference paper at ICLR 2020 Xiang Jiang, Mohammad Havaei, Gabriel Chartrand, Hassan Chouaib, Thomas Vincent, Andrew Jesson, Nicolas Chapados, and Stan Matwin. Attentive task-agnostic meta-learning for few-shot text classification,

    work page 2020

  6. [11]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H ´erve J ´egou, and Tomas Mikolov

    URL https://openreview.net/forum?id=SyxMWh09KX. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H ´erve J ´egou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651,

    arXiv

  7. [12]

    Convolutional neural networks for sentence classification

    Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882,

    arXiv

  8. [16]

    Reptile: a scalable metalearning algorithm

    Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2,

    arXiv

Show all 37 references
  1. [18]

    Linguistically- informed self-attention for semantic role labeling

    Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. Linguistically- informed self-attention for semantic role labeling. arXiv preprint arXiv:1804.08199,

    arXiv

  2. [19]

    Meta-transfer learning for few-shot learning

    Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. arXiv preprint arXiv:1812.02391,

    arXiv

  3. [20]

    Learning to transfer.arXiv preprint arXiv:1708.05629,

    Ying Wei, Yu Zhang, and Qiang Yang. Learning to transfer.arXiv preprint arXiv:1708.05629,

    arXiv

  4. [21]

    Huggingface’s trans- formers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s trans- formers: State-of-the-art natural language processing. ArXiv, abs/1910.03771,

    1910 arXiv

  5. [22]

    Diverse few-shot text classification with multiple metrics.arXiv preprint arXiv:1805.07513,

    Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse few-shot text classification with multiple metrics.arXiv preprint arXiv:1805.07513,

    arXiv

  6. [23]

    Improving few- shot text classification via pretrained language representations

    Ningyu Zhang, Zhanlin Sun, Shumin Deng, Jiaoyan Chen, and Huajun Chen. Improving few- shot text classification via pretrained language representations. arXiv preprint arXiv:1908.08788,

    1908 arXiv

  7. [24]

    Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Processing, pp. 35–45, Copenhagen, Denmark, September

    2017

  8. [25]

    doi: 10.18653/v1/D17-1004

    Association for Computational Linguistics. doi: 10.18653/v1/D17-1004. URL https://www.aclweb.org/anthology/D17-1004. 13 Published as a conference paper at ICLR 2020 A S UPPLEMENTAL MATERIAL A.1 R EGULARIZED LINEAR CLASSIFIER Given anN-wayK-shot classification task, the goal of ...

    2020 doi

  9. [27]

    Implementation details may be found in Appendix A.11

    and P-MAML (Zhang et al., 2019). Implementation details may be found in Appendix A.11. 8https://spacy.io/ 15 Published as a conference paper at ICLR 2020 Method 20 News Amazon HuffPost RCV1 Reuters FewRel 1 shot 5 shot 1 shot 5 shot 1 shot 5 shot 1 shot 5 shot 1 shot 5 shot 1 ...

    2019

  10. [28]

    and P-MAML (Zhang et al., 2019). We run P-MAML on text classification datasets with shorter documents, for which it is feasible to finetune BERT (after WordPiece tokenization, documents from ”longer” datasets exceed BERT’s max length of 512 tokens). 0 2000 4000 # tasks seen duri...

    2019

  11. [30]

    original

    and HuffPost (Figure 12). We observe that our model produces bet- ter separation than the unweighted average AVG and directly using the distributional statistics, s(x) or t(x). For instance, in 20 Newsgroups, our model recognizes three clusters: {talk.religion.misc, soc.religi...

    2020

  12. [32]

    77.1 on 5- way 5-shot, 60.1 vs

    Overall, RR and LR produce similar results (78.0 vs. 77.1 on 5- way 5-shot, 60.1 vs. 58.5 on 5-way 1-shot, respectively), though RR performs slightly better. When using LR as the downstream predictor, learning with distributional signatures improves 5-way 1-shot accuracy by 6....

    2020

  13. [33]

    Since examples in HuffPost Headlines are 30 times shorter, the cosine similarities are higher in this corpora

    This oracle is estimated using all labeled examples from the N classes. Since examples in HuffPost Headlines are 30 times shorter, the cosine similarities are higher in this corpora. AVG: uniform distribution over the words. s(·): word importance estimated directly by s(·). t(...

    2016

  14. [34]

    and induction networks (Geng et al., 2019). Prototypical networks To augment a prototypical network with features learned from distributional signatures, we can construct per-class prototypes based on the attention-weighted representation φ(x) (Section 4.2). From Table 7, we s...

    2019

  15. [36]

    For pretraining, we used Hugging Face’s language model finetuning code with default hyperparameters and BERT’s pretrained base-uncased model

    and MAML (Finn et al., 2017). For pretraining, we used Hugging Face’s language model finetuning code with default hyperparameters and BERT’s pretrained base-uncased model. We applied early stopping when the validation perplexity failed to decrease for 2 epochs. After pre- train...

    2017

  16. [37]

    During the MAML inner loop (adaptation stage), we perform ten updates with step size 10−3

    to make predictions. During the MAML inner loop (adaptation stage), we perform ten updates with step size 10−3. Following Zhang et al. (2019), we do not consider higher order gradients. During the MAML outerloop, we average the gradient across ten sampled tasks and use Adam wi...

    2019

  17. [300]

    MAML MAML meta-learns an initialization such that the model can quickly adapt to new tasks after a few gradient steps

    We apply dropout with rate 0.1 to the hidden layer. MAML MAML meta-learns an initialization such that the model can quickly adapt to new tasks after a few gradient steps. For prediction on the input representation, we use a MLP with one hidden layer of 300 ReLU units. We apply...

    2019

  18. [500]

    s(·): weighted average of word embeddings with weights given by s(·)

    AVG: average word embeddings. s(·): weighted average of word embeddings with weights given by s(·). t(·): weighted average of word embeddings with weights given by t(·). OUR: weighted average of word embeddings with weights given by the attention generator meta-trained on a di...

    2019

  19. [1972]

    Dropout: a simple way to prevent neural networks from overfitting

    12 Published as a conference paper at ICLR 2020 Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958,

    2020

  20. [1997]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146,

    arXiv

  21. [2004]

    A structured self-attentive sentence embedding

    Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130,

    arXiv

  22. [2008]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    arXiv

  23. [2013]

    • P-MAML combines pre-training with MAML

    to predict the relation between each query example and the class prototypes. • P-MAML combines pre-training with MAML. It first finetunes pre-trained BERT repre- sentation on the meta-training data using masked language modeling (Devlin et al., 2018). Based on this finetuned repr...

    2017

  24. [2014]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

  25. [2015]

    Multi-task sequence to sequence learning

    Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114,

    arXiv

  26. [2016]

    Deriving machine attention from human rationales

    Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1903–1913,

    2018

  27. [2017]

    Few-shot learning with graph neural networks

    Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043,

    arXiv

  28. [2018]

    Multi-source domain adaptation with mixture of experts

    Jiang Guo, Darsh Shah, and Regina Barzilay. Multi-source domain adaptation with mixture of experts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4694–4703,

    2018

  29. [2019]

    Meta-learning for low- resource neural machine translation

    Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. Meta-learning for low- resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3622–3631,

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.