{"paper":{"title":"Gaussian Error Linear Units (GELUs)","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"The GELU activation xΦ(x) outperforms ReLU and ELU on computer vision, natural language processing, and speech tasks.","cross_cats":[],"primary_cat":"cs.LG","authors_text":"Dan Hendrycks, Kevin Gimpel","submitted_at":"2016-06-27T19:20:40Z","abstract_excerpt":"We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\\Phi(x)$, where $\\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks."},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the observed gains on the specific tasks and models tested will generalize to other architectures, datasets, and training regimes without further tuning.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"GELU activation xΦ(x) outperforms ReLU and ELU on computer vision, NLP, and speech tasks by weighting inputs by value rather than gating by sign.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"The GELU activation xΦ(x) outperforms ReLU and ELU on computer vision, natural language processing, and speech tasks.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"b83c2b8b1ed504cbde17bb7710d5cbb0ff0e9d2c4eb1d879968aa3d31e368a13"},"source":{"id":"1606.08415","kind":"arxiv","version":5},"verdict":{"id":"7a5657e6-2360-44ec-b2a4-020dcf342710","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-10T12:12:07.972046Z","strongest_claim":"We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.","one_line_summary":"GELU activation xΦ(x) outperforms ReLU and ELU on computer vision, NLP, and speech tasks by weighting inputs by value rather than gating by sign.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the observed gains on the specific tasks and models tested will generalize to other architectures, datasets, and training regimes without further tuning.","pith_extraction_headline":"The GELU activation xΦ(x) outperforms ReLU and ELU on computer vision, natural language processing, and speech tasks."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/1606.08415/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":50,"sample":[{"doi":"","year":null,"title":"Adaptive dropout for training deep neural networks , year =","work_id":"826fb3d9-a3e6-4eaf-8754-cdde878e5a0b","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Learning with pseudo-ensembles , year =","work_id":"5686e296-a575-4d9c-b51e-ca063964598c","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"International Conference on Learning Representations , title =","work_id":"af8adfb7-c005-46f0-a112-dc4457e4627e","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"A Simple Approximation to the Area Under Standard Normal Curve , year =","work_id":"5fdadacd-9a9c-446c-bede-d35471e9e8f4","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Natural Neural Networks , year =","work_id":"517a3f62-3ec1-4077-9263-5464291823a4","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":50,"snapshot_sha256":"181f21b5bd42e8b39f59516aa9d95e98a8e4cca8b1a14087d794a3f29c3cbf43","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"9f3f89fe5fe484d3df62e5d29159aa44c63eedb310a386bdfed1b199af3b5c83"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}