REVIEW 22 cited by
Query-Key Normalization for Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNorm, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply $\ell_2$ normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15.
Forward citations
Cited by 22 Pith papers
-
One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
Different low-precision errors converge on the same query-key spectral runaway, entry is gated by temporal sign-coherence, and a dormant query-key normalization guard contains it.
-
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...
-
Characterization and Mitigation of Training Instabilities in Microscaling Formats
Block-scaled MX low-precision training is unstable because quantization of tightly clustered layer-norm weights and some activations injects multiplicative gradient bias, and this can be fixed by keeping activations i...
-
Absolute Coordinates Make Motion Generation Easy
Using absolute 3D joint coordinates with a plain Transformer and velocity-prediction diffusion outperforms the standard local-relative motion representation, improving fidelity and enabling direct control.
-
Sona Technical Report
Sona, a single model that both generates and ranks recommendations from one user representation, beat a production cascade of more than 15 components in a live Yandex Music A/B test.
-
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Token similarity in Post-Norm decoders is amplified by causal attention at initialization, and RMSNorm backward contraction prevents gradients from repairing the resulting collapse.
-
Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation
Surface keypoint trajectories plus a spatio-temporal contact distance field let one generative model synthesize human motion for single, multiple, and articulated objects without specifying joint types.
-
Raven: High-Recall Sequence Modeling with Sparse Memory Routing
Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.
-
GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation
A causal transformer with 3D RoPE generates vector-quantized 3D Gaussian latent grids autoregressively, enabling unconditional synthesis, completion, and open-ended outpainting of indoor scenes.
-
Multi-view Pyramid Transformer: Look Coarser to See Broader
MVP uses a two-level hierarchy of attention windows and token resolutions to reconstruct large 3D scenes from up to 256 input views in a single feed-forward pass, beating Long-LRM and iLRM on DL3DV and several zero-sh...
-
FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies
A compact 950-million-parameter robot policy trained in about 200 GPU-hours matches or beats multi-billion-parameter baselines on most manipulation benchmarks, including a new best score on CALVIN ABC.
-
iLRM: An Iterative Large 3D Reconstruction Model
iLRM reconstructs 3D Gaussian scenes from multiple photos through iterative refinement of viewpoint tokens, achieving higher quality and speed than prior feed-forward models.
-
NeoBabel: A Multilingual Open Tower for Visual Generation
A 2B multilingual text-to-image model trained on 124M translated pairs matches or beats larger English-only baselines on English while scoring higher on the authors' multilingual benchmark extensions.
-
Towards Foundation Models for Experimental Readout Systems Combining Discrete and Continuous Data
A single transformer with separate pixel and time vocabularies generates realistic Cherenkov detector hits and supports particle identification and noise filtering after fine-tuning.
-
Fast Text-to-Audio Generation with Adversarial Post-Training
ARC post-training speeds up text-to-audio generation to near-real-time speeds on GPUs and a few seconds on phones, without distillation or classifier-free guidance.
-
RayZer: A Self-supervised Large View Synthesis Model
A self-supervised transformer model predicts camera poses and scene features from unposed images and renders novel views, reaching performance on par with pose-supervised baselines.
-
Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Streaming DiLoCo trains billion-parameter LLMs at data-parallel quality while cutting the inter-datacenter bandwidth by about two orders of magnitude via partial, overlapped, and 4-bit-quantized synchronization.
-
DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
A 1B-parameter autoregressive world model generates over 40 seconds of controllable driving video with near-SOTA FVD, but key claims rest on inconsistent and cross-paper comparisons.
-
NeuralPLexer3: Accurate Biomolecular Complex Structure Prediction with Flow Models
A flow-based model, NeuralPLexer3, predicts biomolecular complex structures with improved physical validity and speed over AlphaFold3 on the PoseBusters benchmark, and introduces NPBench and ConfBench for broader evaluation.
-
SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training
SnapGen is a 379M-parameter UNet with cross-architecture distillation and a 1.38M-parameter decoder that generates 1024x1024 images on a phone in about 1.4 seconds, with GenEval 0.66 and ImageNet FID 2.06.
-
BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics
BCAT, a block causal transformer for next-frame prediction, achieves state-of-the-art accuracy on 2D fluid dynamics PDE benchmarks, beating larger foundation models with fewer parameters.
-
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
Discussion (0). Continue with ORCID to comment.