Self-Balanced Dropout

Chenhao Su; Renfen Hu; Shen Li; Zhengdong Lu

REVIEW 3 major objections 3 minor 29 references

Self-Balanced Dropout

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Self-Balanced Dropout claims a trainable replacement value can cancel the co-adaptation that dropout leaves behind.

desk verdict The central derivation in Section 3 is algebraically invalid, and the experiments are too thin to carry the empirical claim; it should not be reviewed in current form. read the letter →

arxiv 1908.01968 v1 pith:KIL5F7BO submitted 2019-08-06 cs.CL cs.LG

Shen Li , Chenhao Su , Renfen Hu , Zhengdong Lu This is my paper · ORCID

classification cs.CLcs.LG

keywords dropoutco-adaptationregularizationtrainablemaskinputcorrelationnaturallanguageprocessingtextclassificationnamedentityrecognition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard dropout only partially prevents co-adaptation: when input features are correlated, the dropout regularizer pushes weights in correlated directions, which reproduces the overfitting it was meant to cure. It introduces Self-Balanced Dropout, which replaces each dropped unit with a trainable scalar or vector rather than with zero, and derives a regularized objective in which the replacement value can grow to offset the influence of large correlated inputs on parameter updates. The authors report consistent improvements over strong baselines on seven sentence-classification datasets, two named-entity-recognition benchmarks, and WMT 2014 English-German translation. If the claim holds, the method is a drop-in replacement for dropout that needs no architectural change and no extra tuning.

What carries the argument

The central object is the trainable replacement variable $x_{\text{mask}}$: a scalar when the input unit is a value, a vector when the input is a word embedding. In the paper's derivation, Eq. (8) writes the expected loss of randomly keeping $x_{ij}$ or replacing it with $x_{\text{mask}}$, and Eq. (9) is the reduced objective $Q(w)+\hat{R}(w)$ with $Q(w)=\|y-(1-p)X_{\text{mask}}w\|^2$ and $\hat{R}(w)=p(1-p)\sum_i\sum_j [(x_{ij}+x_{\text{mask}})w_j]^2$. The claimed mechanism is that the $(x_{ij}+x_{\text{mask}})^2$ factor dilutes the influence of correlation among the $x_{ij}$, while the $Q(w)$ term drives an inverse relation between $w$ and $x_{\text{mask}}$. All experimental results are read as confirming that this balancing effect appears during training.

What would settle it

Evaluate Eq. (8) and Eq. (9) on a one-dimensional linear regression with fixed values for $y$, $x$, $x_{\text{mask}}$, $p$, and $w$ by expanding both sides numerically; if the two sides are unequal, the paper's formal reduction is not valid.

Watch

Extended reading notes

Core claim

The paper's central claim is that dropout's regularization term $\hat{R}(w)=\frac{1-p}{p}\sum_i \sum_j (x_{ij}w_j)^2$ makes parameter updates depend on $x_{ij}^2$, so correlated features push the corresponding weights in similar directions and co-adaptation persists. The discovery is that replacing a dropped unit with a trainable value $x_{\text{mask}}$ changes the regularizer into a form where the gradient with respect to $w_j$ contains $(x_{ij}+x_{\text{mask}})^2 w_j$; a large learned $x_{\text{mask}}$ therefore balances the differences among correlated input values and steers updates away from pure input correlation. The paper also claims an additional term forces $w$ and $x_{\text{mask}}$ into an inverse relationship, letting the mask grow as weights shrink. On this basis the authors assert that Self-Balanced Dropout alleviates the co-adaptation problem that original dropout leaves unresolved.

Load-bearing premise

The theoretical argument hinges on an algebraic identity that rewrites the expected masked loss as the sum of two squared terms plus a weighted square; if that identity fails for some inputs, the claimed balancing effect of the trainable replacement value is not a proven consequence.

Editorial extensions

If this is right

Self-Balanced Dropout can replace standard dropout in any layer, including hidden layers, without changing the network architecture.
The method should help most when input features are strongly correlated, which the paper ties to word co-occurrence in NLP.
During training the mask value should grow while the ratio of weight norm to input norm falls, matching the paper's reported curves.
The derivation reframes masked-replacement pretraining as a principled form of balanced dropout rather than a purely empirical trick.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A controlled regression experiment with tunable feature correlation could test whether the gain scales with correlation strength, a prediction the paper implies but does not directly measure.
Per-feature or per-layer masks may extend the idea further; the paper notes that sharing a single mask across all dimensions gave no statistically significant gain.
If the inverse $w$ vs. $x_{\text{mask}}$ relationship is the active mechanism, then adding an explicit regularizer or schedule on the mask might stabilize training even more, which the paper leaves unexamined.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the central theoretical claim is wrong. Expanding Eq. (8) correctly gives ||y − (pX + (1−p)Xmask)w||^2 + p(1−p) Σ_i Σ_j (x_ij − x_mask)^2 w_j^2, not the three-term expression in Eq. (9). The variance term has the wrong sign and the mean term is split in a way that manufactures a nonexistent cross term. A one-dimensional check (y=1, x=2, xmask=3, p=0.5, w=1) gives 2.5 versus their 6.5. Since Eq. (11) and the balancing story both rest on this reduction, the theory does not stand.

That is a pity, because the basic trick is reasonable. Replacing dropped units with a trainable value instead of zero is simple and easy to implement, and extending it to hidden layers is a modest extension of the BERT/denoising masking idea that the paper explicitly cites. The experiments cover classification, NER, and MT, the code is released, and the NER tables report error bars. The authors also honestly note in Section 4.2 that masking multiple dimensions gives no statistically significant improvement. That candor is real.

The soft spots are proportionate to the claim. The classification table has no error bars and uses per-dataset tuned keep probabilities, so gains like MR 81.5→81.7 are noise-level. The MT gain is 0.2 BLEU. The observed growth of xmask in Figure 3 is not independent evidence because xmask is trained on the same data. And there is no comparison to the closest baselines—word dropout with a fixed unknown-token embedding, or BERT-style masking—so we do not know whether trainability of the mask is what helps.

As a heuristic, a practitioner could try it. As a research paper, the load-bearing derivation is invalid and the empirical case is not strong enough to compensate. I would not cite the theoretical part, and I would not send this to review in its current form. If the authors fix the algebra and reframe the method as an empirical variant with proper significance testing, it might become a useful workshop paper. As submitted, reject.

Referee Report

3 major / 3 minor

Summary. The paper argues that standard dropout leaves a residual co-adaptation problem when input features are correlated, and that this follows from the regularizer that dropout induces in linear regression. To fix it, the authors propose Self-Balanced Dropout, which replaces dropped units with a trainable mask value x_mask instead of zero. They derive an expected-loss objective in Section 3, claim it decomposes into terms that force an inverse relationship between weights and x_mask, and report accuracy/F1/BLEU improvements over baselines on sentence classification, named entity recognition, and machine translation. The central theoretical step is the reduction of Eq. (8) to Eq. (9).

Significance. If the derivation were correct, Self-Balanced Dropout would be a simple and broadly applicable dropout variant with a concrete theoretical motivation, and the paper's multi-task evaluation and released source code would make it a useful contribution. However, the central derivation is algebraically incorrect, and the experimental results are not sufficiently strong or well-controlled to stand on their own. The contribution as stated is therefore not supported.

major comments (3)

[Section 3, Eqs. (8)-(9)] The reduction of the expected loss is algebraically incorrect. For independent Bernoulli masks, the exact identity is E_R[||y - ((R◦X)+(I-R)◦Xmask)w||^2] = ||y - (pX + (1-p)Xmask)w||^2 + p(1-p) Σ_i Σ_j (x_ij - x_mask)^2 w_j^2. The paper instead claims it equals ||y - pXw||^2 + ||y - (1-p)Xmask w||^2 + p(1-p) Σ_i Σ_j [(x_ij + x_mask)w]^2. These expressions are not equal: the mean term is a single squared norm of the expected prediction error, not a sum of two squared norms, and the variance term uses (x_ij - x_mask)^2, not (x_ij + x_mask)^2. A one-dimensional check with y=1, x=2, x_mask=3, p=0.5, w=1 gives 2.5 for the correct expression and 6.5 for Eq. (9). Since Eq. (11), the claimed inverse relationship between w and x_mask, and the balancing interpretation all derive from the incorrect Eq. (9), the theoretical motivation for Self-Balanced Dropout is unsupported.
[Section 4, Tables 1-4] The experimental evidence is too weak to compensate for the invalid derivation. Table 1 reports single accuracies without error bars or significance tests, and most improvements over CNN-non-static are under one percentage point. Tables 2 and 3 report F1 means with standard deviations that overlap between baseline and Self-Balanced Dropout (e.g., 90.32 ± 0.26 vs. 90.73 ± 0.25 on CoNLL-2003). The paper itself states in Section 4.2 that applying the method across multiple dimensions 'does not seem statistically significant.' Thus the claim of consistent, significant improvement across tasks is not established.
[Section 4.1.3, Table 4] The Transformer comparison is confounded: the Self-Balanced Dropout model is trained with a dropout rate 0.05 smaller than the baseline, so the reported BLEU gain (27.3 vs. 27.5) cannot be attributed to the proposed mechanism. No significance or variance information is given for the BLEU scores, and the result appears to be a single run.

minor comments (3)

[Throughout] There are several typographical errors, including 'brifely' (Section 2), 'datesets' (Section 4.1.1), 'exsit' (Section 4.1.1), and inconsistent spelling of 'co-adaption' versus 'co-adaptation.'
[Section 3, Eq. (10)] The notation X_mask is defined as an N×D matrix in which every entry is the same trainable scalar, which is clear from context, but the reader must infer that the scalar is shared only within a single application layer; Section 4.2 later says sharing across dimensions is not beneficial, so the exact sharing scheme should be stated precisely in the method.
[Section 4.2, Figure 3] Figure 3 would be more informative with error bars or multiple runs, and the caption does not state the initialization of x_mask or how norm(w)/norm(X) is computed, which makes the claimed trend hard to evaluate.

Circularity Check

1 steps flagged · score 2.0 of 10

One fitted-parameter confirmation is mildly circular, but the core held-out accuracy claims remain independent.

fitted input called prediction [Section 4.2, Figure 3 and surrounding text]
"In order to understand where the improvement comes from and to verify that our calculation is correct, we further analyze the changes in the norm ofX,Xmask andw in the training time. Figure 3 shows the changes of norm(w)/norm(X) and Xmask in two experiments. After several epochs, thenorm(w)/norm(X) continues to decrease, which means the regularization termR(w) is more determined by X. Meanwhile, Xmask keeps increasing as expected, thus diminishes the inﬂuence of correlation in X on parameter updates."

Xmask is the trainable parameter introduced by the method and optimized on the same training data through the objective in Eq. (8)/(9). Its observed growth during training is therefore a property of the fitted model, not an independent confirmation of the theory. Presenting the trained mask's trajectory as 'as expected' verification turns the fitted parameter itself into evidence for the mechanism it was designed to implement. This is a mild circularity: the behavior being confirmed is generated by the very parameter the theory is supposed to explain. It is not load-bearing for the central empirical claim, which is held-out accuracy, so it does not by itself invalidate the experimental comparisons.

full rationale

The central derivation chain in Section 3 is not circular in the sense of defining the conclusion into the inputs: Self-Balanced Dropout is defined through a trainable x_mask, and the algebraic steps leading to Eq. (9) are mathematical manipulations of that definition. However, the reduction from Eq. (8) to Eq. (9) is algebraically incorrect, as the correct expansion of the expectation yields a single squared mean term and a variance term proportional to (x_ij - x_mask)^2, not the two separate squared norms and (x_ij + x_mask)^2 shown in Eq. (9). This is a soundness/correctness problem, not a circularity problem, so it does not raise the circularity score by itself. The one genuinely circular element is the use of Figure 3 as confirmation: x_mask is a trained parameter, and saying that 'Xmask keeps increasing as expected' and using this to 'verify that our calculation is correct' treats the fitted parameter's trajectory as an independent prediction. That is a fitted-input-called-prediction pattern, but it is peripheral. The paper's main evidence, the accuracy/F1/BLEU comparisons on held-out data, is externally measured and not reduced to the training objective. Self-citations are present in the references, but none is load-bearing for the central claim. The paper itself notes in Section 4.2 that applying the method in multiple dimensions 'does not seem statistically significant,' and several reported improvements lack error bars or significance tests; those are limitations and correctness-risk concerns, not circularity. On balance, the derivation chain is not fundamentally circular, but the Figure 3 confirmation is a minor circular step. Score 2 reflects that one peripheral fitted-confirmation issue while acknowledging that the core held-out empirical claims are independent.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The theoretical contribution rests on an algebraic step that fails under direct expansion. The method introduces trainable mask parameters and per-dataset dropout probabilities that are fit to the data. No new structural entities are introduced.

free parameters (4)

x_mask = trained during optimization
A trainable scalar or vector that replaces dropped units; its final value is not reported, only the norm trajectory in Figure 3.
p1 = per-dataset values, e.g., 0.8 for MR, 0.9 for SST-1, 1.0 for SST-2, 0.5 for Subj, 0.8 for TREC, 0.9 for CR, 0.9 for MPQA
Input-layer keep probability, chosen separately for each task in Table 1, so the reported gains may reflect tuned hyperparameters.
p2 = per-dataset values, e.g., 0.4 for MR, 0.7 for SST-1, 0.6 for SST-2, 0.6 for Subj, 0.6 for TREC, 0.6 for CR, 0.4 for MPQA
Hidden-layer keep probability, co-tuned with p1 for each dataset.
Transformer dropout rate reduction = 0.05 lower than baseline
The paper changes the dropout rate for the proposed method on the translation task, which is a hyperparameter adjustment beyond the baseline configuration.

assumptions (4)

ad hoc to paper The expected loss under Self-Balanced Dropout equals Eq. (9).
Asserted in Section 3; direct expansion contradicts it.
domain assumption Linear regression dropout analysis transfers to deep networks and their hidden layers.
Section 2's motivation uses a linear model, then Section 3 asserts the same co-adaptation mechanism applies at any layer without proof.
domain assumption A single trainable x_mask can balance the influence of all correlated input features.
The balancing argument treats x_mask as a large additive constant that offsets x, but a single scalar cannot match the varying values of different features.
standard math SGD with a fixed learning rate is the update rule; Eq. (5) captures the parameter dynamics.
Standard optimization setup used to discuss update directions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Self-Balanced Dropout." pith.science (2026). https://pith.science/paper/KIL5F7BO

@misc{pith2026190801968,
  author       = {Pith},
  title        = {Pith review of: Self-Balanced Dropout},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KIL5F7BO}},
  note         = {Machine review of arXiv:1908.01968}
}

read the original abstract

Dropout is known as an effective way to reduce overfitting via preventing co-adaptations of units. In this paper, we theoretically prove that the co-adaptation problem still exists after using dropout due to the correlations among the inputs. Based on the proof, we further propose Self-Balanced Dropout, a novel dropout method which uses a trainable variable to balance the influence of the input correlation on parameter update. We evaluate Self-Balanced Dropout on a range of tasks with both simple and complex models. The experimental results show that the mechanism can effectively solve the co-adaption problem to some extent and significantly improve the performance on all tasks.

Figures

Figures reproduced from arXiv: 1908.01968 by the authors.

**Figure 2.** mask with a trainable vector 4.1 Datasets and Experiment Settings 4.1.1 Sentence Classification We employ the same seven datesets with (Kim, 2014), including both sentiment analysis and topic classification tasks. MR: Movie reviews sentiment datasets (Pang and Lee, 2005). SST1: Stanford Sentiment Treebank with 5 sentiment labels (Socher et al., 2013). To keep same with (Kim, 2014), we train the model on both phra… view at source ↗

**Figure 3.** Changes in the kWk/kXk and ||Xmask|| during training in two experiments [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 10 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Pierre Baldi and Peter J Sadowski. 2013. Understanding dropout. In Advances in neural information processing systems, pages 2814--2822

work page 2013
[4]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018
[5]

John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis

1957
[6]

David P Helmbold and Philip M Long. 2015. On the inductive bias of dropout. The Journal of Machine Learning Research, 16(1):3403--3454

work page 2015
[7]

Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580

arXiv 2012
[8]

Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90\ In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 57--60. Association for Computational Linguistics

work page 2006

Show all 29 references

[9]

Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168--177. ACM

2004
[10]

Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882

2014 arXiv
[11]

Shen Li, Zhe Zhao, Tao Liu, Renfen Hu, and Xiaoyong Du. 2017. Initializing convolutional filters with semantic features for text classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1884--1889

2017
[12]

Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1--7. Association for Computational Linguistics

2002
[13]

Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics, page 271. Association for Computational Linguistics

2004
[14]

Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 115--124. Association for Computational Linguistics

2005
[15]

Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Bj \"o rkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning,...

2013
[16]

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language p...

2013
[17]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929--1958

2014
[18]

Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. arXiv preprint arXiv:1702.02098

2017 arXiv
[19]

Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142--147. Association for Computa...

2003
[20]

Gomez, Stephan Gouws, Llion Jones, ukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit

Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, ukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. http://arxiv.org/abs/1803.07416 Tensor2tensor for neural machine transl...

2018 arXiv
[21]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998--6008

2017
[22]

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096--1103. ACM

2008
[23]

Stefan Wager, Sida Wang, and Percy S Liang. 2013. Dropout training as adaptive regularization. In Advances in neural information processing systems, pages 351--359

2013
[24]

Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165--210

2005
[25]

Wenpeng Yin and Hinrich Sch \"u tze. 2016. Multichannel variable-size convolution for sentence classification. arXiv preprint arXiv:1603.04513

2016 arXiv
[26]

Rui Zhang, Honglak Lee, and Dragomir Radev. 2016 a . Dependency sensitive convolutional neural networks for modeling sentences and documents. arXiv preprint arXiv:1611.02361

2016 arXiv
[27]

Ye Zhang, Stephen Roller, and Byron Wallace. 2016 b . Mgnc-cnn: A simple approach to exploiting multiple word embeddings for sentence classification. arXiv preprint arXiv:1603.00968

2016 arXiv
[28]

Ye Zhang and Byron Wallace. 2015. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820

2015 arXiv
[29]

Rui Zhao and Kezhi Mao. 2017. Topic-aware deep compositional models for sentence classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(2):248--260

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Pierre Baldi and Peter J Sadowski. 2013. Understanding dropout. In Advances in neural information processing systems, pages 2814--2822

work page 2013

[4] [4]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018

[5] [5]

John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis

1957

[6] [6]

David P Helmbold and Philip M Long. 2015. On the inductive bias of dropout. The Journal of Machine Learning Research, 16(1):3403--3454

work page 2015

[7] [7]

Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580

arXiv 2012

[8] [8]

Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90\ In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 57--60. Association for Computational Linguistics

work page 2006