{"work":{"id":"3a43a02d-e005-47ad-8373-c166e20c9ee9","openalex_id":null,"doi":null,"arxiv_id":"1710.05941","raw_key":null,"title":"Searching for Activation Functions","authors":null,"authors_text":"Prajit Ramachandran, Barret Zoph, Quoc V. Le","year":2017,"venue":"cs.NE","abstract":"The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, $f(x) = x \\cdot \\text{sigmoid}(\\beta x)$, which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9\\% for Mobile NASNet-A and 0.6\\% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.","external_url":"https://arxiv.org/abs/1710.05941","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-25T04:40:23.580845+00:00","pith_arxiv_id":"1710.05941","created_at":"2026-05-09T05:45:21.267958+00:00","updated_at":"2026-06-05T21:23:00.469572+00:00","title_quality_ok":false,"display_title":"Searching for Activation Functions","render_title":"Searching for Activation Functions"},"hub":{"state":{"work_id":"3a43a02d-e005-47ad-8373-c166e20c9ee9","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":63,"external_cited_by_count":null,"distinct_field_count":22,"first_pith_cited_at":"2019-05-28T17:05:32+00:00","last_pith_cited_at":"2026-05-22T08:46:21+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-06-06T10:00:43.492842+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":12},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":9},{"context_polarity":"unclear","n":3},{"context_polarity":"use_method","n":1}],"runs":{"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T18:09:40.617201+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"Gaussian Error Linear Units (GELUs)","work_id":"0466fd22-03a1-4a61-af0a-a900e77bb023","shared_citers":15},{"title":"Layer Normalization","work_id":"20a2d720-0046-4c7c-bcd6-327ec8143f69","shared_citers":8},{"title":"GLU Variants Improve Transformer","work_id":"17d0763c-1016-41ab-a478-478e890765eb","shared_citers":7},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":7},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":6},{"title":"Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289","work_id":"619b409a-ddb8-4401-b55e-d8ca367322ce","shared_citers":4},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":4},{"title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","work_id":"c888e6d1-0b1d-43d6-9ef5-f0912a0efa1b","shared_citers":4},{"title":"Mish: A self regularized non- monotonic neural activation function","work_id":"f314c3b6-cec7-4246-8006-cbed6b9840d3","shared_citers":4},{"title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","work_id":"4e5eee26-cd04-4c7a-988f-3e6d1a1f0eb9","shared_citers":4},{"title":"Adam: A Method for Stochastic Optimization","work_id":"1910796d-9b52-4683-bf5c-de9632c1028b","shared_citers":3},{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","shared_citers":3},{"title":"BLOOM: A 176B-Parameter Open-Access Multilingual Language Model","work_id":"337ba690-f35d-4154-9450-8edf4bc9f488","shared_citers":3},{"title":"Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling","work_id":"c7f2f5a9-ae4b-48db-aff0-24b9d0528995","shared_citers":3},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":3},{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","work_id":"4ee75248-1199-492c-a52f-6661e0f4adff","shared_citers":3},{"title":"Mixtral of Experts","work_id":"0de8c352-9daa-4e1e-8c7b-3d0dec69f369","shared_citers":3},{"title":"Retentive Network: A Successor to Transformer for Large Language Models","work_id":"5b0449ac-92b0-41f2-8b4f-586c2b5a08b6","shared_citers":3},{"title":"RWKV: Reinventing RNNs for the Transformer Era","work_id":"524dc80d-f4ef-4f89-bf1a-9a8c1e4b6a81","shared_citers":3},{"title":"S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S","work_id":"168a55d5-675d-49cf-be47-a17ee8cd742e","shared_citers":3},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":3},{"title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","work_id":"9b10667a-da61-4358-aceb-10578234d45d","shared_citers":3},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":3},{"title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","work_id":"1c4b4409-c14b-488b-a086-c57a5aab8a29","shared_citers":3}],"time_series":[{"n":2,"year":2020},{"n":2,"year":2023},{"n":2,"year":2024},{"n":1,"year":2025},{"n":27,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T18:10:12.620425+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T18:10:02.415202+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"Searching for Activation Functions","claims":[{"claim_text":"The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Searching for Activation Functions because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T18:09:48.819534+00:00"}},"summary":{"title":"Searching for Activation Functions","claims":[{"claim_text":"The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Searching for Activation Functions because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"Gaussian Error Linear Units (GELUs)","work_id":"0466fd22-03a1-4a61-af0a-a900e77bb023","shared_citers":15},{"title":"Layer Normalization","work_id":"20a2d720-0046-4c7c-bcd6-327ec8143f69","shared_citers":8},{"title":"GLU Variants Improve Transformer","work_id":"17d0763c-1016-41ab-a478-478e890765eb","shared_citers":7},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":7},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":6},{"title":"Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289","work_id":"619b409a-ddb8-4401-b55e-d8ca367322ce","shared_citers":4},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":4},{"title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","work_id":"c888e6d1-0b1d-43d6-9ef5-f0912a0efa1b","shared_citers":4},{"title":"Mish: A self regularized non- monotonic neural activation function","work_id":"f314c3b6-cec7-4246-8006-cbed6b9840d3","shared_citers":4},{"title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","work_id":"4e5eee26-cd04-4c7a-988f-3e6d1a1f0eb9","shared_citers":4},{"title":"Adam: A Method for Stochastic Optimization","work_id":"1910796d-9b52-4683-bf5c-de9632c1028b","shared_citers":3},{"title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","work_id":"e96730e3-129b-4db6-b981-15ab7932e297","shared_citers":3},{"title":"BLOOM: A 176B-Parameter Open-Access Multilingual Language Model","work_id":"337ba690-f35d-4154-9450-8edf4bc9f488","shared_citers":3},{"title":"Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling","work_id":"c7f2f5a9-ae4b-48db-aff0-24b9d0528995","shared_citers":3},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":3},{"title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","work_id":"4ee75248-1199-492c-a52f-6661e0f4adff","shared_citers":3},{"title":"Mixtral of Experts","work_id":"0de8c352-9daa-4e1e-8c7b-3d0dec69f369","shared_citers":3},{"title":"Retentive Network: A Successor to Transformer for Large Language Models","work_id":"5b0449ac-92b0-41f2-8b4f-586c2b5a08b6","shared_citers":3},{"title":"RWKV: Reinventing RNNs for the Transformer Era","work_id":"524dc80d-f4ef-4f89-bf1a-9a8c1e4b6a81","shared_citers":3},{"title":"S., Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S","work_id":"168a55d5-675d-49cf-be47-a17ee8cd742e","shared_citers":3},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":3},{"title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","work_id":"9b10667a-da61-4358-aceb-10578234d45d","shared_citers":3},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":3},{"title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","work_id":"1c4b4409-c14b-488b-a086-c57a5aab8a29","shared_citers":3}],"time_series":[{"n":2,"year":2020},{"n":2,"year":2023},{"n":2,"year":2024},{"n":1,"year":2025},{"n":27,"year":2026}],"dependency_candidates":[]},"authors":[]}}