REVIEW 6 cited by
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
read the original abstract
Recent advancements in large language models (LLMs) based on transformer architectures have sparked significant interest in understanding their inner workings. In this paper, we introduce a novel approach to modeling transformer architectures using highly flexible non-autonomous neural ordinary differential equations (ODEs). Our proposed model parameterizes all weights of attention and feed-forward blocks through neural networks, expressing these weights as functions of a continuous layer index. Through spectral analysis of the model's dynamics, we uncover an increase in eigenvalue magnitude that challenges the weight-sharing assumption prevalent in existing theoretical studies. We also leverage the Lyapunov exponent to examine token-level sensitivity, enhancing model interpretability. Our neural ODE transformer demonstrates performance comparable to or better than vanilla transformers across various configurations and datasets, while offering flexible fine-tuning capabilities that can adapt to different architectural constraints.
Forward citations
Cited by 6 Pith papers
-
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...
-
DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformers
DynImmune-BERT models longitudinal TCR repertoires as continuous-time, event-aware trajectories and reports higher AUC than static baselines on lung and thyroid cancer classification.
-
DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformers
DynImmune-BERT shows that event-aware continuous-time modeling of longitudinal TCR repertoires improves cancer-status AUC over static and simpler temporal baselines, but external validation is limited by small cohorts.
-
The Transformer as a Polar State Estimator
Transformer components arise as the natural solution to precision-weighted directional state estimation on the hypersphere.
-
The Transformer as a Polar State Estimator
The standard Transformer block arises as a first-order approximation to a polar state estimator on the hypersphere, with a Polar Transformer retaining higher-order terms.
-
The Transformer as a Polar State Estimator
The paper casts the standard Transformer block with RoPE as a first-order approximation of a radial–tangential state estimator and introduces a Polar Transformer variant that retains the discarded geometric corrections.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.