Kolmogorov-Arnold Transformer
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:CL6AOOAYrecord.jsonopen to challenge →
read the original abstract
Transformers stand as the cornerstone of mordern deep learning. Traditionally, these models rely on multi-layer perceptron (MLP) layers to mix the information between channels. In this paper, we introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces MLP layers with Kolmogorov-Arnold Network (KAN) layers to enhance the expressiveness and performance of the model. Integrating KANs into transformers, however, is no easy feat, especially when scaled up. Specifically, we identify three key challenges: (C1) Base function. The standard B-spline function used in KANs is not optimized for parallel computing on modern hardware, resulting in slower inference speeds. (C2) Parameter and Computation Inefficiency. KAN requires a unique function for each input-output pair, making the computation extremely large. (C3) Weight initialization. The initialization of weights in KANs is particularly challenging due to their learnable activation functions, which are critical for achieving convergence in deep neural networks. To overcome the aforementioned challenges, we propose three key solutions: (S1) Rational basis. We replace B-spline functions with rational functions to improve compatibility with modern GPUs. By implementing this in CUDA, we achieve faster computations. (S2) Group KAN. We share the activation weights through a group of neurons, to reduce the computational load without sacrificing performance. (S3) Variance-preserving initialization. We carefully initialize the activation weights to make sure that the activation variance is maintained across layers. With these designs, KAT scales effectively and readily outperforms traditional MLP-based transformers.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
KANs need curvature: penalties for compositional smoothness
A curvature penalty for KANs, derived to respect compositional effects and equipped with a proven upper bound on full-model curvature, produces smoother activations while preserving accuracy.
-
From Zero to Detail: A Progressive Spectral Decoupling Paradigm for UHD Image Restoration with New Benchmark
A new framework called ERR decomposes UHD image restoration into three frequency stages with specialized sub-networks and introduces the LSUHDIR benchmark dataset of over 82,000 images.
-
Tensor-based Multi-layer Decoupling
A new tensor framework for multi-layer decoupling of multivariate functions is proposed via ParaTuck decompositions and bilevel optimization.
-
KAN-MLP-Mixer: A comprehensive investigation of the usage of Kolmogorov-Arnold Networks (KANs) for improving IMU-based Human Activity Recognition
A hybrid KAN-MLP model for IMU-based human activity recognition achieves 5.33% relative macro F1 improvement over pure MLPs on eight datasets by placing KANs at input embedding and classification stages.
-
Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection
AAND is a two-stage anomaly detection method that advances a pre-trained teacher via residual anomaly amplification and applies hard knowledge distillation in reverse distillation to achieve SOTA results on MVTecAD, V...
-
KAN-MLP-Mixer: A comprehensive investigation of the usage of Kolmogorov-Arnold Networks (KANs) for improving IMU-based Human Activity Recognition
A hybrid KAN-MLP architecture with KAN input embedding and specialized LarctanKAN classification layer yields 5.33% average macro F1 gain over pure-MLP baselines in IMU-based human activity recognition.
-
FMC-DETR: Frequency-Decoupled Multi-Domain Coordination for Aerial-View Object Detection
FMC-DETR proposes a frequency-decoupled fusion framework with WeKat backbone, MDFC coordination, and CPF fusion modules that claims state-of-the-art results on remote sensing object detection benchmarks.
-
P1-KAN: an effective Kolmogorov-Arnold network with application to hydraulic valley optimization
P1-KAN introduces a new KAN architecture with theoretical approximation guarantees that outperforms MLPs and prior KAN variants on irregular functions while matching spline KAN accuracy on smooth ones, demonstrated on...
-
A Practitioner's Guide to Kolmogorov-Arnold Networks
A systematic review of Kolmogorov-Arnold Networks that maps their relation to Kolmogorov superposition theory, MLPs, and kernels, examines basis-function design choices, summarizes performance advances, and supplies a...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.