Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Visualizing and Understanding the Effectiveness of BERT

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that pre-training's benefit is geometric: it places BERT at a good initial point that leads to wider optima and easier optimization during fine-tuning compared with training from scratch.

desk verdict Useful geometric story for BERT fine-tuning, but the trajectory projection does not actually project onto the plotted surfaces; the layer-rollback analysis is the strongest evidence. read the letter →

arxiv 1908.05620 v1 pith:542YWXAC submitted 2019-08-15 cs.CL cs.LG

Yaru Hao , Li Dong , Furu Wei , Ke Xu This is my paper · ORCID
classification cs.CLcs.LG
keywords BERTpre-trainingfine-tuninglosslandscapeoptimizationtrajectoryflatminimageneralizationtransferability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to explain why pre-training-then-fine-tuning beats training from scratch for BERT on downstream NLP tasks. Using visualizations of loss landscapes and optimization trajectories, it claims that pre-training reaches a good initial point across tasks, which leads to wider optima and easier optimization. It also claims that fine-tuned BERT generalizes better because its optima are flat and wide, and because the training loss surface lines up with the generalization error surface. The paper further claims that BERT's lower layers stay more invariant during fine-tuning, meaning they encode transferable language representations. If these claims hold, the practical lesson is that pre-training's value is largely geometric: it puts the model in a region where optimization is smooth and the final solution is stable under perturbation.

What carries the argument

The central machinery is loss-landscape visualization adapted from neural-network visualization work: 1D loss curves along the interpolation between initialization and final parameters; 2D loss surfaces spanned by the fine-tuning direction on the target dataset and the fine-tuning direction on another dataset (with the claim that these are divergent and orthogonal); and projected optimization trajectories computed by decomposing each epoch's parameter movement into components along these two axes. A separate layer-rollback procedure restricts the perturbation to layer groups (0-7, 8-15, 16-23) to inspect per-layer transferability. These tools turn the high-dimensional optimization of a 345M-parameter model into pictures that support claims about optimum width, smoothness of descent, robustness to overfitting, and layer-wise invariance.

What would settle it

Measure the actual cosine similarity between the target-dataset optimization direction and the other-dataset direction; if it is not near zero, the second axis is not orthogonal and the 2D surfaces are distorted slices, not a faithful plane. Alternatively, compute the exact orthogonal projection of each epoch's parameter change onto the delta_1-delta_2 plane and compare it with the method of Equation (5); large discrepancies would invalidate the plotted trajectories.

Watch

Extended reading notes

Core claim

The central discovery is a set of geometric facts about the loss landscape of BERT fine-tuning. On four GLUE-style datasets (MNLI, RTE, SST-2, MRPC), the two-dimensional training loss surfaces around fine-tuned BERT show noticeably wider, flatter optima than the same architecture trained from random initialization. The optimization trajectories from the pre-trained start point are smoother and more direct, converging faster and reaching lower final training loss, while from-scratch trajectories are rougher and can cross obstacles. The generalization error surface is consistent with the training loss surface for fine-tuned models, so the wide optima correspond to regions of small development-set error. Finally, rollback experiments show that restoring the lower 0th-7th layers of fine-tuned BERT to their pre-trained values barely hurts accuracy, while rolling back the upper 16th-23rd layers collapses performance; the loss surfaces restricted to lower layers are wide, and those restricted to higher layers are sharp. The paper reads these observations as evidence that pre-training provides a good initial point, that fine-tuning is robust to overfitting even with 345M parameters, and that lower layers learn transferable representations.

Load-bearing premise

The load-bearing premise is that the 2D loss surfaces and the projected trajectories faithfully represent the real optimization dynamics, in particular that the two chosen axes are genuinely orthogonal and that the trajectory points are true projections onto that plane.

Editorial extensions

If this is right

  • If pre-training's benefit is geometric, then the same architecture trained from scratch with a better initialization scheme or a smoother loss surface should close part of the gap to BERT fine-tuning.
  • Fine-tuning algorithms that explicitly seek flat and wide optima, such as weight averaging or entropy-based regularization, should show larger gains when starting from a pre-trained point than from random initialization.
  • Because lower layers are nearly invariant during fine-tuning, freezing or only lightly updating them should preserve most downstream accuracy while saving compute.
  • The consistency between training loss and generalization error surfaces on small datasets predicts that fine-tuned BERT will be robust to extended training, which the paper confirms by fine-tuning for 5 plus 20 epochs on MRPC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to plot the same loss surfaces for other pre-trained encoders such as GPT or ELMo; the geometric account predicts wider optima than from-scratch training, with the width gap growing with model size.
  • The layer-rollback evidence suggests a simple diagnostic for transferability: the width of the layer-restricted loss surface around the fine-tuned point predicts how much that layer group can be reused across tasks without retraining.
  • If the flat-minima account is correct, pre-training should also make downstream fine-tuning more robust to hyperparameter choice and label noise, since wide optima tolerate perturbations; this can be checked by measuring accuracy variance across seeds and learning rates.
  • The paper's orthogonality assertion for its two visualization axes is left unquantified, so a fair reader should verify the cosine similarity before relying on the 2D surfaces.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper visualizes the loss landscapes and fine-tuning optimization trajectories of BERT-large on four GLUE-style tasks (MNLI, RTE, SST-2, MRPC), comparing pre-training-then-fine-tuning with training from scratch. It reports three main findings: pre-training supplies an initialization that leads to wider optima and easier optimization; fine-tuning BERT is robust to overfitting; and lower BERT layers are more invariant and transferable across tasks. The evidence consists of 1D loss curves, 2D loss and generalization-error surfaces, projected optimization trajectories, learning curves, and layer-rollback experiments.

Significance. If the claims were fully supported, the paper would provide a useful geometric account of why pre-training-then-fine-tuning outperforms training from scratch in NLP, connecting flat/wide optima to generalization for an over-parameterized model, and offering a simple rollback diagnostic for layer transferability. The layer-wise rollback experiments and the train/generalization surface comparisons are concrete and falsifiable, and the paper is careful to note that all conclusions are derived from BERT fine-tuning. However, the trajectory-projection methodology in Section 3.3 is flawed as written: the plotted trajectories are not coordinates in the displayed δ1/δ2 plane, so the visual evidence for smoother optimization and overfitting robustness currently lacks support. This is a fixable methodological issue rather than a fatal one, because the learning curves, 1D slices, and rollback table are independent evidence.

major comments (3)
  1. [Section 3.3, Eq. (5)] Equation (5) computes dβ_i as the square root of the squared Euclidean distance from δ_i to the span of δ1 after normalization; this quantity is nonnegative, carries no sign, and does not reference δ2. Consequently (dα_i,dβ_i) is not a coordinate in the plane spanned by δ1 and δ2, and the trajectories overlaid on Figures 3 and 4 are not paths on the displayed loss or error surfaces. This undermines the visual evidence in Section 5.2 for smoother optimization and in Section 5.3 for overfitting robustness. Please replace Equation (5) with a signed projection onto δ2 (for example, dβ_i = (δ_i · δ2)/||δ1||^2 after the normalization described in Section 3.2), re-plot the trajectories, and verify that they lie on the displayed surfaces; alternatively, state explicitly that only distances from the δ1 axis are shown and adjust the interpretation accordingly.
  2. [Section 3.2] The assertion that δ1 and δ2 are 'divergent and orthogonal' is not supported by any reported cosine similarity, angle, or other numerical evidence. If the angle between the two axes deviates substantially from 90 degrees, the 2D surfaces in Figures 1, 3, 4, and 6 use oblique coordinates, and visual comparisons of 'wider' optima between fine-tuning and training from scratch are distorted. Please report the cosine or angle for every dataset-axis pair used, and either restrict the visualization to nearly orthogonal pairs or orthogonalize the axes (for example, by Gram-Schmidt) before plotting.
  3. [Section 5.3 and Figure 4] The overfitting-robustness conclusion is drawn from a single fine-tuning run on MRPC, and the text does not report quantitative accuracy or error values for the 5+20 epoch run or a comparison with a scratch-trained model under the same extended schedule. Because this is one of the paper's three headline findings, please add the development-set numbers across epochs and, ideally, multiple random seeds or additional small datasets; otherwise, narrow the claim to a qualitative illustration rather than a general robustness result.
minor comments (4)
  1. [Section 3.3, Eq. (3)] The symbol × is used for what must be a dot product, since the formula produces a scalar cosine; the cross product is not defined in the parameter-space dimension of BERT. Please replace × with · in Equations (3) and (4).
  2. [Section 3.1] There is a duplicated article in 'θ0 represents the the pre-trained parameters'; please correct the typo.
  3. [Section 7, Table 1] The statement that rollbacking middle layers 'does not dramatically decrease' performance is hard to reconcile with the MRPC row, where rolling back layers 8–15 lowers accuracy by 10.05 points; please qualify this claim with the per-dataset magnitudes.
  4. [Experimental Setup] No random seeds, number of repetitions, or code are provided, so the reader cannot assess run-to-run variance in the loss-surface and trajectory figures; please add reproducibility details.

Circularity Check

0 steps flagged · score 0.0 of 10

No self-definitional or fitted-input circularity; the loss-surface visualization is an empirical measurement, not a derivation that assumes its conclusions.

full rationale

The paper's central claims are empirical observations drawn from loss-landscape visualizations, learning curves, and rollback experiments, not from a derivation whose conclusion is equivalent to an input by construction. The 1D/2D loss curves are computed by interpolating between the initialized and fine-tuned parameters, and the comparison between fine-tuning BERT and training from scratch is a direct measurement of the geometry; no fitted parameter is later renamed as a prediction. The flatness-generalization link is imported from prior work (Hochreiter and Schmidhuber, Keskar et al., Li et al.) as an external assumption, which is not circular. The paper invokes no uniqueness theorem, and the only overlapping-author citation (Dong et al. 2019 in the introduction) is incidental background, not load-bearing. The trajectory-projection method in Section 3.3 is questionable as a mathematical procedure—Equation (5) yields only a nonnegative perpendicular magnitude and does not locate a signed coordinate in the δ1/δ2 plane—but this is a validity concern, not circularity, because the conclusion is not made equivalent to the input by construction. Therefore no circular step is identified, and the paper should not receive a circularity penalty beyond zero.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities; it relies on standard loss-landscape visualization assumptions and one dataset-specific modeling choice (δ2 from another dataset). The main risk is whether the chosen directions faithfully represent optimization and generalization geometry.

assumptions (5)
  • domain assumption Flatness and width of a local optimum correlate with generalization performance.
    Borrowed from Hochreiter and Schmidhuber 1997, Keskar et al. 2016, and Li et al. 2018; used in Sections 5.1 and 6.1 to interpret visual width as evidence of generalization.
  • domain assumption Linear interpolation between initial and fine-tuned parameters is a meaningful probe of the loss landscape.
    Equations (1) and (2) define loss curves along θ0 to θ1; this follows Goodfellow and Vinyals 2015, but assumes straight-line sections capture optimization-relevant geometry.
  • ad hoc to paper The second axis δ2 (fine-tuning direction on another dataset) forms a near-orthogonal 2D subspace with δ1.
    Section 3.2 asserts orthogonality without quantitative support; if false, the 2D loss surfaces are distorted slices.
  • domain assumption Development set accuracy is a sufficient proxy for generalization error.
    Generalization surfaces in Section 6.2 are computed on the dev set only; no test set or variance estimate is provided.
  • ad hoc to paper Global norm normalization of direction vectors preserves geometric comparisons across models.
    Section 3.2 re-scales δ2 to the norm of δ1; unlike per-filter normalization in Li et al. 2018, this may not be scale invariant and could bias flatness comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visualizing and Understanding the Effectiveness of BERT." pith.science (2026). https://pith.science/paper/542YWXAC

@misc{pith2026190805620,
  author       = {Pith},
  title        = {Pith review of: Visualizing and Understanding the Effectiveness of BERT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/542YWXAC}},
  note         = {Machine review of arXiv:1908.05620}
}
read the original abstract

Language model pre-training, such as BERT, has achieved remarkable results in many NLP tasks. However, it is unclear why the pre-training-then-fine-tuning paradigm can improve performance and generalization capability across different tasks. In this paper, we propose to visualize loss landscapes and optimization trajectories of fine-tuning BERT on specific datasets. First, we find that pre-training reaches a good initial point across downstream tasks, which leads to wider optima and easier optimization compared with training from scratch. We also demonstrate that the fine-tuning procedure is robust to overfitting, even though BERT is highly over-parameterized for downstream tasks. Second, the visualization results indicate that fine-tuning BERT tends to generalize better because of the flat and wide optima, and the consistency between the training loss surface and the generalization error surface. Third, the lower layers of BERT are more invariant during fine-tuning, which suggests that the layers that are close to input learn more transferable representations of language.

Figures

Figures reproduced from arXiv: 1908.05620 by the authors.

Figure 1
Figure 1. Training loss surfaces of training from scratch (top) and fine-tuning BERT (bottom) on four datasets. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Training loss of fine-tuning BERT and train [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. The optimization trajectory of fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: One-dimensional training loss curves. Dash [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Two-dimensional generalization error surfaces of training from scratch (top) and fine-tuning BERT [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Layer-wise training loss surfaces on the MNLI dataset (top) and the MRPC dataset (bottom). The dot [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TRACE for Tracking the Emergence of Semantic Representations in Transformers

    cs.CL 2025-05 reject novelty 6.0 of 10

    Using Hessian curvature, intrinsic dimensionality, and linguistic probes on a synthetic frame-semantic corpus, the paper claims a coordinated intersection-based phase transition in small transformers, though the marke...

  2. Module-Aware Parameter-Efficient Machine Unlearning on Transformers

    cs.LG 2025-08 conditional novelty 5.0 of 10

    MAPE-Unlearn uses Fisher-information-based scores and greedy search to select important heads and filters, then applies sparse unlearning updates, claiming improved efficacy-fidelity trade-offs on Transformers.

  3. Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond

    cs.LG 2025-02 conditional novelty 5.0 of 10

    Adding sharpness-aware minimization to NPO unlearning slows relearning attacks on WMDP and MUSE benchmarks.

Reference graph

Works this paper leans on

37 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785

    arXiv 2019

  4. [4]

    Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment

    work page 2006

  5. [5]

    Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC’09

    work page 2009

  6. [6]

    Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. 2017. https://arxiv.org/pdf/1611.01838.pdf Entropy- SGD : Biasing gradient descent into wide valleys

    arXiv 2017

  7. [7]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. https://doi.org/10.1007/11736790_9 The pascal recognising textual entailment challenge . In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW'05, pages 177--190, Berl...

    doi:10.1007/11736790_9 2006

  8. [8]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805

    arXiv 2018

Show all 37 references
  1. [9]

    William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005)

    2005

  2. [10]

    Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197

    2019 arXiv

  3. [11]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. https://www.aclweb.org/anthology/W07-1401 The third PASCAL recognizing textual entailment challenge . In Proceedings of the ACL - PASCAL Workshop on Textual Entailment and Paraphrasing , pages 1--9, Prague....

    2007

  4. [12]

    Yoav Goldberg. 2019. http://arxiv.org/abs/1901.05287 Assessing BERT 's syntactic abilities . CoRR, abs/1901.05287

    2019 arXiv

  5. [13]

    Goodfellow and Oriol Vinyals

    Ian J. Goodfellow and Oriol Vinyals. 2015. http://arxiv.org/abs/1412.6544 Qualitatively characterizing neural network optimization problems . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

    2015 arXiv

  6. [14]

    Sepp Hochreiter and J\" u rgen Schmidhuber. 1997. https://doi.org/10.1162/neco.1997.9.1.1 Flat minima . Neural Comput., 9(1):1--42

    1997 doi

  7. [15]

    Jeremy Howard and Sebastian Ruder. 2018 a . https://www.aclweb.org/anthology/P18-1031 Universal language model fine-tuning for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328--3...

    2018

  8. [16]

    Jeremy Howard and Sebastian Ruder. 2018 b . http://arxiv.org/abs/1801.06146 Universal language model fine-tuning for text classification . In ACL. Association for Computational Linguistics

    2018 arXiv

  9. [17]

    Daniel Jiwoong Im, Michael Tao, and Kristin Branson. 2016. http://arxiv.org/abs/1612.04010 An empirical analysis of deep network loss surfaces . CoRR, abs/1612.04010

    2016 arXiv

  10. [18]

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407

    2018 arXiv

  11. [19]

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. http://arxiv.org/abs/1609.04836 On large-batch training for deep learning: Generalization gap and sharp minima . CoRR, abs/1609.04836

    2016 arXiv

  12. [20]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. http://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations, San Diego, CA

    2015 arXiv

  13. [21]

    Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. https://www.aclweb.org/anthology/P18-1132 LSTM s can learn syntax-sensitive dependencies well, but modeling structure makes them better . In Proceedings of the 56th Annual Meeting of...

    2018

  14. [22]

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems

    2018

  15. [23]

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. http://arxiv.org/abs/1611.01368 Assessing the ability of LSTM s to learn syntax-sensitive dependencies . CoRR, abs/1611.01368

    2016 arXiv

  16. [24]

    Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A

    Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. 2019 a . http://arxiv.org/abs/1903.08855 Linguistic knowledge and transferability of contextual representations . CoRR, abs/1903.08855

    2019 arXiv

  17. [25]

    Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019 b . http://arxiv.org/abs/1901.11504 Multi-task deep neural networks for natural language understanding . CoRR, abs/1901.11504

    2019 arXiv

  18. [26]

    Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6297--6308

    2017

  19. [27]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. http://www.aclweb.org/anthology/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North American Chapter of the As...

    2018

  20. [28]

    Matthew Peters, Sebastian Ruder, and Noah A. Smith. 2019. https://arxiv.org/pdf/1903.05987.pdf To tune or not to tune? Adapting pretrained representations to diverse tasks

    2019 arXiv

  21. [29]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/language-unsupervised/language understanding paper.pdf Improving language understanding by generative pre-training

    2018

  22. [30]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  23. [31]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://www.aclweb.org/anthology/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conferenc...

    2013

  24. [32]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019 a . https://arxiv.org/abs/1905.05950 BERT rediscovers the classical NLP pipeline . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

    2019 arXiv

  25. [33]

    Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019 b . https://openreview.net/forum?id=SJzSgnRcKX What do you learn from context? P robing for sentence structur...

    2019

  26. [34]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In Advances in Neural Information Processing Systems 30,...

    2017

  27. [35]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://openreview.net/forum?id=rJ4km2R5t7 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In International Conference on Learning Representations

    2019

  28. [36]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  29. [37]

    Chiyuan Zhang, Samy Bengio, and Yoram Singer. 2019. https://arxiv.org/abs/1902.01996 Are all layers created equal? arXiv preprint arXiv:1902.01996

    2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.