REVIEW 36 cited by
Differential Transformer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling noise. Specifically, the differential attention mechanism calculates attention scores as the difference between two separate softmax attention maps. The subtraction cancels noise, promoting the emergence of sparse attention patterns. Experimental results on language modeling show that Diff Transformer outperforms Transformer in various settings of scaling up model size and training tokens. More intriguingly, it offers notable advantages in practical applications, such as long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and reduction of activation outliers. By being less distracted by irrelevant context, Diff Transformer can mitigate hallucination in question answering and text summarization. For in-context learning, Diff Transformer not only enhances accuracy but is also more robust to order permutation, which was considered as a chronic robustness issue. The results position Diff Transformer as a highly effective and promising architecture to advance large language models.
Forward citations
Cited by 36 Pith papers
-
PithTrain: A Compact and Agent-Native MoE Training System
PithTrain is a compact agent-native MoE training system that matches production throughput and improves agent-task efficiency by up to 62% fewer turns and 64% less GPU time on the new ATE-Bench.
-
Pay Less Attention to Function Words for Free Robustness of Vision-Language Models
FDA differentially subtracts function-word cross-attention from original attention heads to cut attack success rates by 18-90% across models and tasks while dropping performance by at most 0.6%.
-
IAFormer: Interaction-Aware Transformer network for collider data analysis
IAFormer uses boost-invariant pairwise quantities and differential attention to create a sparse Transformer that achieves state-of-the-art classification on top-quark and quark-gluon jet datasets while using over an o...
-
Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.
-
RTL-Sequencer: Towards Scalable RTL Timing Prediction with the Sequence-based Paradigm
Linearizing RTL logic cones into breadth-first sequences and processing them with Mamba-2 sequence models yields better arrival-time, WNS, and TNS predictions than graph-based baselines on 21 open-source designs.
-
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex s...
-
HADT: A Heterogeneous Multi-Agent Differential Transformer for Autonomous Earth Observation Satellite Cluster
Proposes HADT, a heterogeneous multi-agent differential transformer with relational observations-actions tokenization for model-free RL-based autonomous resource management in EO satellite clusters, claiming gains ove...
-
ROVER: Routing Object-Centric Visual Evidence for Grounded Multi-Image Reasoning
ROVER introduces a learnable routing plugin for object-centric visual evidence in MLLMs via token triplets and differential attention, reporting gains on MM-GCoT and VideoEspresso when integrated into Qwen2.5-VL-7B.
-
Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
Empirical update to prior work shows most of 20 recent Transformer modifications do not transfer at 1-3B scales when measured with downstream CLIMB-12 tasks, multi-seed noise floor, and cross-scale stability.
-
The General Theory of Localization Methods
The localization method unifies kernel methods, local learning algorithms, MeanShift, Hopfield networks, and Transformers through local models, localization tricks, and hierarchical extensions.
-
Beyond Similarity: Temporal Operator Attention for Time Series Analysis
TOA augments attention with learnable sequence-space operators and stochastic regularization to enable signed temporal mixing, yielding gains on forecasting and related benchmarks when added to PatchTST and iTransformer.
-
FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
FocuSFT uses an inner optimization loop to adapt fast-weight parameters into a parametric memory that sharpens attention on relevant content, then conditions outer-loop supervised fine-tuning on this representation, y...
-
SAGE: Selective Attention-Guided Extraction for Token-Efficient Document Indexing
SAGE is a training-free context reduction method that converts attention signals from a small LLM into a differential relevance heatmap to select top units for downstream QA, achieving competitive accuracy at 10% toke...
-
Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
Brain-OF is a multimodal foundation model for fMRI, EEG and MEG using any-resolution sampling, DINT attention with sparse MoE, and masked temporal-frequency pretraining on ~40 datasets to achieve superior downstream p...
-
Degradation-Aware Metric Prompting for Hyperspectral Image Restoration
A single hyperspectral-restoration model that computes six interpretable input statistics (frequency, texture, spectral-curvature metrics) and uses them as prompts to route Mixture-of-Experts modules achieves state-of...
-
Multi-view Pyramid Transformer: Look Coarser to See Broader
MVP uses a two-level hierarchy of attention windows and token resolutions to reconstruct large 3D scenes from up to 256 input views in a single feed-forward pass, beating Long-LRM and iLRM on DL3DV and several zero-sh...
-
Accelerated Machine Learning Force Field for Predicting Thermal Conductivity of Organic Liquids
A single machine-learned force field predicts thermal conductivity of 20 organic liquids with ~14% MAPE by aligning simulated density to experiments.
-
Flashlight: PyTorch Compiler Extensions to Accelerate Attention Variants
Flashlight is a compiler-native PyTorch framework that generates efficient fused kernels for arbitrary and data-dependent attention variants, supporting more cases than FlexAttention with competitive performance.
-
LLM-Enhanced Multi-Agent Reinforcement Learning with Expert Workflow for Real-Time P2P Energy Trading
An LLM-enhanced MARL system with differential attention critic produces lower economic costs and voltage violations than baselines in simulated real-time P2P electricity trading.
-
Differential Amplifier-Inspired AmpAttention for Multi-View Robotic Manipulation
AmpAttention and RVAF raise multi-view robotic manipulation success and cut training time by suppressing attention noise with a differential-amplifier-style mechanism plus a CMRR loss.
-
SamatNext v0.2-B: An Exploratory Study of RMS-Normalized Hybrid Decoders for Curriculum Retention in Small Code Models
SamatNext v0.2-B reaches 100% on Stage 5 and retains 98.8% of Stage 3 behavior versus 97.6% and 6% for the Transformer baseline in a controlled curriculum setting.
-
SamatNext v0.2-B: An Exploratory Study of RMS-Normalized Hybrid Decoders for Curriculum Retention in Small Code Models
SamatNext v0.2-B reaches 100% on Stage 5 while retaining 98.8% of Stage 3 behavior, versus 97.6% and 6% for a matched Transformer baseline, in a controlled curriculum setting.
-
Building The Ph(ysical)AI Layer Of Machine Intelligence
A principle-driven RF encoder achieves 77.7% average accuracy across 15 cross-modal tasks, performing better on physically grounded tasks than semantic ones.
-
Beyond Similarity: Temporal Operator Attention for Time Series Analysis
Temporal Operator Attention augments softmax attention with learnable sequence-space operators for signed temporal mixing and uses stochastic regularization to enable practical training, yielding consistent gains on t...
-
When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
DyT improves validation loss 27% at 64M params/1M tokens but worsens it 19% at 118M tokens, with saturation levels predicting the sign of the effect.
-
Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data
CLMM is a two-stage contrastive learning framework using CNN-DiffTransformer encoders and dual-branch fusion to improve multimodal human activity recognition under limited labels.
-
GraphDiffMed: Knowledge-Constrained Differential Attention with Pharmacological Graph Priors for Medication Recommendation
GraphDiffMed integrates dual-scale differential attention with pharmacological graph priors to improve medication recommendation quality, ranking, and safety balance on MIMIC-III data.
-
CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language Models
A training method that injects token-level causal labels into attention improves out-of-distribution accuracy on a synthetic benchmark and slightly on math/reasoning tasks.
-
Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation
X-Agent adds agent tokens, chosen by optimal-transport affinity between text and visual keys, to CLIP attention, reporting marginal mIoU gains (0.1-0.6%) over prior OVSS methods.
-
EIFNet: Leveraging Event-Image Fusion for Robust Semantic Segmentation
EIFNet fuses event and image features with three new attention modules and reports state-of-the-art segmentation accuracy on DDD17 and DSEC-Semantic.
-
Visible-Light Imaging Diagnosis of Neutral Particle Emission Tomography in the Tokamak Divertor: An Efficient Transformer-based Surrogate Model
A transformer-based network predicts 2D neutral-particle emission distributions in the EAST tokamak divertor from consecutive visible-light frames, achieving roughly 20 FPS with accuracy comparable to slower tomograph...
-
Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
This work systematically compares inter-layer and intra-layer hybridization strategies for combining self-attention and Mamba-style state space models, evaluating them on language modeling, downstream tasks, long-cont...
-
DiffLoRA: Differential Low-Rank Adapters for Large Language Models
DiffLoRA, a low-rank adapter variant of differential attention, generally underperforms LoRA but shows gains on HumanEval and multi-value needle retrieval.
-
Bridging the Gap Between Natural Language and Market Dynamics via High-Dimensional Representation Learning
Siamese-optimized high-dimensional FinBERT embeddings outperform scalar sentiment baselines and raw embeddings for short-term stock price prediction on the FNSPID dataset.
-
The General Theory of Localization Methods
The localization method is presented as a unifying framework connecting kernel methods, MeanShift, Hopfield networks, LLE, fuzzy inference, denoising autoencoders, and Transformers via local models and the localization trick.
-
A Cellular Doctrine of Morality: Intrinsic Active Precision and the Mind-Reality Overload Dilemma
AI incorporating active precision from pyramidal neurons may reduce information overload by evaluating evidence coherence before attention rather than maximizing rewards.
Discussion (0). Sign in to comment.