Of Non-Linearity and Commutativity in BERT

Damian Pascual; Gino Brunner; Roger Wattenhofer; Sumu Zhao

RECORD

Of Non-Linearity and Commutativity in BERT

Sumu Zhao, Damian Pascual, Gino Brunner, Roger Wattenhofer

Computation and Language · Submitted 12 Jan 2021

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections.

how it is stated · hedged possibility 0.68 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,509 input tokens.

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.14 · human participants 0.02 · controlled comparison 0.19 · randomized assignment 0.01 · states quantitative result 0.42 · compares with prior results 0.29 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v4. Results and the main claim are underlined; show every role.

In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-linearity of different elements of transformers. Next, we focus our investigation on the feed-forward networks (FFN) inside transformers, which contain 2/3 of the model parameters and have so far not received much attention. We find that FFNs are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance. Moreover, we study the interactions between layers in BERT and show that, while the layers exhibit some hierarchical structure, they extract features in a fuzzy manner. Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections. This provides a justification for the strong performance of recurrent and weight-shared transformer models.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT.	contribution 0.71	0.77	0.02	asserted 0.79
s1	First, we propose a method to measure the degree of non-linearity of different elements of transformers.	contribution 0.62	0.93	0.03	asserted 0.94
s2	Next, we focus our investigation on the feed-forward networks (FFN) inside transformers, which contain 2/3 of the model parameters and have so far not received much attention.	contribution 0.68	0.25	0.03	asserted 0.95
s3	We find that FFNs are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance.	result 1.00	0.96	0.20	asserted 1.00
s4	Moreover, we study the interactions between layers in BERT and show that, while the layers exhibit some hierarchical structure, they extract features in a fuzzy manner.	result 0.81	0.95	0.07	asserted 1.00
s5	Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections.	result 0.64	0.92	0.43	hedged possibility 0.68
s6	This provides a justification for the strong performance of recurrent and weight-shared transformer models.	interpretation 1.00	0.65	0.15	asserted 0.85

What kind of paper the classifier reports this is: empirical study 0.89 · evidence from observation or experiment 0.92.

What was examined, and what was not

sources read

arXiv listing abstract v4

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

In this work we provide new insights into the transformer architecture, and in particular, its best-known variant, BERT. First, we propose a method to measure the degree of non-linearity of different elements of transformers. Next, we focus our investigation on the feed-forward networks (FFN) inside transformers, which contain 2/3 of the model parameters and have so far not received much attention. We find that FFNs are an inefficient yet important architectural element and that they cannot simply be replaced by attention blocks without a degradation in performance. Moreover, we study the interactions between layers in BERT and show that, while the layers exhibit some hierarchical structure, they extract features in a fuzzy manner. Our results suggest that BERT has an inductive bias towards layer commutativity, which we find is mainly due to the skip connections. This provides a justification for the strong performance of recurrent and weight-shared transformer models.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

6 arXiv papers cite this in 7 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 5 sentences without a confident label

Cites it as support 1 paper

The motivation for this study is two-fold: * The inputs undergo a non-linear transformation when passing through the activation functions in the feed-forward layers of deep neural networks [ref] . * The parameters in the position-wise feed-forward layers of the transformer account for two-thirds of the total model's parameters ($8d^2$ per layer, d is the model’s hidden dimension).

Interpretability in Activation Space Analysis of Transformers: A Focused Survey · Introduction · 0.99

Cites it as background 1 paper

[ref] quantified the contextual knowledge of BERT and [ref] analyzed the embedding spaces of BERT in order to quantify the non-linearity of its layers.

On Isotropy Calibration of Transformers · Related Work · 1.00

What it cites · 6 sentences without a confident label

Uses the method or tool of 1 paper

We also see that adding a GeLU non-linearity [ref] to the self-attention block further improves performance.

Gaussian Error Linear Units (GELUs) · MLP vs. Self-Attention · 0.87

Cites as background 7 papers

Since the introduction of the transformer architecture [ref] , and in particular of BERT [ref] , researchers have tried to gain a deeper understanding of the inner workings of these models [ref] .

A Primer in BERTology: What we know about how BERT works · Introduction · 1.00
In this field, a number of studies look into attention distributions in order to understand transformers' decisions and their internal dynamics [ref] .

Do Attention Heads in BERT Track Syntactic Dependencies? · Related Work · 0.94
Since then, many different variants of transformers have been proposed [ref] .

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter · Related Work · 1.00

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.