Dual PatchNorm

Manoj Kumar; Mostafa Dehghani; Neil Houlsby

Dual PatchNorm

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.01327 v3 pith:P2QCOYP6 submitted 2023-02-02 cs.CV cs.LG

Dual PatchNorm

Manoj Kumar , Mostafa Dehghani , Neil Houlsby This is my paper

classification cs.CV cs.LG

keywords dualpatchnormlayertransformersvisionaccuracyalternativebefore

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization
cs.CV 2024-12 unverdicted novelty 6.0

DOLLAR combines variational score and consistency distillation for few-step video generation plus latent reward optimization, reporting 82.57 VBench score and up to 278x speedup over the teacher diffusion model for 12...