TND models sequences via independent neuron dynamics on a directed graph and reports over three times more consecutive catches than strong baselines on a Pong behavior-cloning task.
Exploring Sparsity in Recurrent Neural Networks
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
abstract
Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.
verdicts
UNVERDICTED 4representative citing papers
Task-aware pruning improves OOD model performance by realigning distorted OOD layerwise norm and pairwise-distance profiles with the task-adapted geometry observed on ID inputs.
TALE selectively prunes task-detrimental layers in LLMs at inference time to match or exceed baseline performance with lower computational cost across multiple models and tasks.
Simulation study finds embedding network in cortical microcolumn model enhances core information flux through biases and recurrence resonance.
citing papers explorer
-
Topological Neural Dynamics: A Neuron-wise Framework for Sequence Modeling
TND models sequences via independent neuron dynamics on a directed graph and reports over three times more consecutive catches than strong baselines on a Pong behavior-cloning task.
-
TAPIOCA: Why Task- Aware Pruning Improves OOD model Capability
Task-aware pruning improves OOD model performance by realigning distorted OOD layerwise norm and pairwise-distance profiles with the task-adapted geometry observed on ID inputs.
-
TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination
TALE selectively prunes task-detrimental layers in LLMs at inference time to match or exceed baseline performance with lower computational cost across multiple models and tasks.
-
Are cortical microcircuits optimized for information flux? -- A simulation-based reverse engineering study
Simulation study finds embedding network in cortical microcolumn model enhances core information flux through biases and recurrence resonance.