REVIEW 48 cited by
Microscaling Data Formats for Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Microscaling Data Formats for Deep Learning
read the original abstract
Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements. MX formats balance the competing needs of hardware efficiency, model accuracy, and user friction. Empirical results on over two dozen benchmarks demonstrate practicality of MX data formats as a drop-in replacement for baseline FP32 for AI inference and training with low user friction. We also show the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss and no modifications to the training recipe.
Forward citations
Cited by 48 Pith papers
-
Grid Games: The Power of Multiple Grids for Quantizing Large Language Models
Allowing each quantization group to select among multiple 4-bit grids improves accuracy over single-grid FP4 for both post-training and pre-training of LLMs.
-
One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
Different low-precision errors converge on the same query-key spectral runaway, entry is gated by temporal sign-coherence, and a dormant query-key normalization guard contains it.
-
Jack of All Scales: A Versatile FPGA Tensor Block for MXFP Precisions
Targeted DSP tensor-mode changes enable native MXFP4/MXFP6/E4M3 support on Agilex-5-like FPGAs, with a 36% block-area cost and 4.2x average systolic-array throughput gain over baseline mapping strategies.
-
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Full-stack NVFP4 LLM pretraining is stable with a 1.47% loss gap to BF16 via LoRA-SVD projections, transformed AdamW/Root optimizers, and mixed-precision attention.
-
LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation
LongLive-2.0 delivers an NVFP4 parallel infrastructure that enables direct training of long multi-shot autoregressive diffusion video models and achieves up to 2.15x training and 1.84x inference speedups on Blackwell ...
-
Multi-Scale Dequant: Eliminating Dequantization Bottleneck via Activation Decomposition for Efficient LLM Inference
MSD eliminates dequantization from the GEMM path by decomposing BF16 activations into multiple low-precision parts that multiply directly with INT8 or MXFP4 weights, achieving near-16 effective bits for INT8 and 6.6 f...
-
Pretraining large language models with MXFP4 on Native FP4 Hardware
Weight-gradient quantization drives most convergence problems in MXFP4 pretraining of Llama 3.1-8B; deterministic Hadamard rotations stabilize training by correcting structured micro-scaling errors.
-
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Four Over Six adaptively scales blocks in NVFP4 quantization to smaller FP4 values, making representable value distributions more uniform and reducing quantization error especially for near-maximal values.
-
MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention
A closed-form MXFP4 scale boundary Qmax=7.25 plus pre-normalization softmax quantization recovers nearly all FP16 video quality lost by naive MXFP4 attention.
-
dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
dMX learns continuous per-layer MXFP bit-width offsets with temperature annealing and target-aware regularization, yielding Pareto-better mixed-precision LLMs than uniform or KL-heuristic baselines.
-
Studying quantization trade-offs for efficient inference deployment in machine translation
Quantized Hy-MT2 models stay accurate at long context, but quantized EuroLLM 9B/22B models collapse (up to ~60% chrF++ drop) while W4A8/W8A8 plus 200–400-token chunking improves serving throughput.
-
LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference
LightRot uses grouped local rotation plus outlier alignment to make 4-bit LLaMA inference accurate and cheap, claiming 27.4 TOPS/W on a 28nm accelerator.
-
GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference
Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...
-
Stable FP4 Training via Transposition-Invariant Block Quantization
Transposition-invariant 2D block FP4 quantization plus truncation-free scaling and stochastic rounding enables stable end-to-end FP4 LLM training within ~1% of BF16.
-
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.
-
CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits
CANN Bench is an Ascend-NPU benchmark with 53 operators and 1,060 released test cases that scores agent-generated kernels on compilation, correctness, and hardware-anchored performance.
-
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
SharQ combines input-adaptive N:M sparsity and FP4 quantization via sparse backbone plus dense residual, recovering 43-63% of the NVFP4-to-FP16 accuracy gap on Llama and Qwen models without calibration or retraining.
-
HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
HyperQuant unifies Hadamard transform, optimal lattice quantization, and entropy coding to outperform prior schemes on LLM weight and KV cache quantization down to 1.7 bits per scalar while preserving quality on a 19B...
-
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
E2M1 FP4 has inherent shrinkage bias from asymmetric bin geometry that accumulates and destabilizes training; UFP4 with uniform E1M2/INT4 grids and selective RHT/stochastic rounding reduces BF16-relative degradation i...
-
ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling
ReSET mitigates accuracy degradation in NVFP4-quantized reasoning models via step-aware entropy-based temperature scaling and provides a small-M CUDA kernel for up to 2.5x kernel speedup and 2x end-to-end speedup.
-
dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats
dMX is a differentiable mixed-precision framework that learns per-layer MXFP bit-width assignments for LLMs and outperforms KL-based heuristics on perplexity and zero-shot accuracy under bit-width budgets.
-
O-POPE: High-Frequency Pipelined Outer Product based GEMM acceleration with minimal buffering overhead
O-POPE is an outer-product GEMM accelerator that repurposes FPU pipeline registers for buffering to reach 1 GHz in 12 nm FINFET with under 2% buffer area and 99.97% utilization.
-
Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature
A multi-LLM council scores predictive processing papers on an expert ontology, maps results in 3D hypothesis space, and introduces a dispersion metric showing greater spread in global versus local oddball paradigms.
-
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
ThriftAttention recovers 89.1% of the FP16 quality gap versus pure FP4 attention by running only 5% of query-key blocks in FP16 on long-context benchmarks.
-
SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization
SOAR improves NVFP4 post-training quantization accuracy for LLMs by analytically solving joint scale optimization and searching decoupled scales.
-
The Entropy of Floating-Point Numbers
An analytic approximation for floating-point entropy is derived that links to a new quantity, with scale-invariance proven and closed forms given for common distributions.
-
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
LoKA enables practical FP8 use in numerically sensitive large recommendation models via profiling, model adaptations, and runtime kernel orchestration.
-
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
LoKA enables practical FP8 use in numerically sensitive large recommendation models via online profiling of activations, reusable model modifications for stability, and dynamic kernel dispatching.
-
Pretraining large language models with MXFP4 on Native FP4 Hardware
Weight gradient quantization is the main driver of instability in full-pipeline FP4 LLM training, mitigated by deterministic Hadamard rotations rather than added stochasticity.
-
Pretraining large language models with MXFP4 on Native FP4 Hardware
Weight gradient FP4 quantization drives LLM pretraining divergence, which deterministic Hadamard rotations can stabilize on native MXFP4 hardware.
-
LOCALUT: Harnessing Capacity-Computation Tradeoffs for LUT-Based Inference in DRAM-PIM
LOCALUT delivers 1.82x geometric mean speedup for quantized DNN inference on real UPMEM DRAM-PIM devices by using operation-packed LUTs with canonicalization, reordering, and slice streaming.
-
DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
DynamiQ compresses gradients for multi-hop all-reduce with per-block bit-width allocation, block reordering, and fused recompression, matching BF16 accuracy at ~5 bits per coordinate and improving time-to-accuracy by ...
-
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
By appending quantized residual channels to NVFP4 activations and duplicating the matching weights, ARCQuant reaches W4A8-level accuracy while keeping a single unified 4-bit GEMM.
-
SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision
SeVeDo achieves 12.7–13.8 TOPS/W in 28nm for ViT-Base and Llama2-7B by combining SVD-based low-rank decomposition, hierarchical group quantization, and SVD-guided mixed precision.
-
WINT: A Novel Weighted Integer Representation with Improved Error Characteristics
WINT (M imes2^E) with 2–3 exponent bits gives both larger range and lower mean relative error than same-width integers for bit widths of 12 and above.
-
Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding
Cassandra is a self-speculative decoding system that builds a draft model via fine-grained data selection and optimized pruning/mantissa truncation, achieving up to 2.41x speedup over BF16 and 1.81x more tokens than E...
-
MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation
MX-SAFE proposes a versatile MXFP format with on-the-fly bit allocation and tile-based design that reports small accuracy gains over prior MX formats and an accelerator using 24.9% less energy than BF16 while matching...
-
The Thermodynamic Costs of Simple Linear Regression
Thermodynamic lower bounds are approximated for exact and SGD linear regression, producing energy-aware scaling laws for optimal training dataset size given a target generalization error.
-
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
LoKA claims to make FP8 practical for large recommendation models via statistical probing, model adaptations, and accuracy-aware kernel dispatch.
-
StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
Chunked streaming top-k enables CSA indexer execution at 1M sequence length with 6.21 GB peak memory and >=0.998 recall on synthetic V4-shaped inputs.
-
OSC: Hardware Efficient W4A4 Quantization via Outlier Separation in Channel Dimension
OSC separates token-persistent outlier channels in activations into a compact high-precision tensor for dual-path 4-bit GEMM computation, limiting accuracy loss to roughly 1-2 points on Qwen3 models while delivering u...
-
Diagonal-Tiled Mixed-Precision Attention for Efficient Low-Bit MXFP Inference
DMA delivers a fused low-bit MXFP attention kernel with diagonal tiling that achieves significant speedup on B200 GPUs with negligible generation quality loss.
-
Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
PLENA introduces a co-designed system with three optimization pathways for long-context agentic LLM inference, claiming up to 2.23x throughput over A100 and 4.04x energy efficiency.
-
GoldenFloat: A Phi-Derived Static-Split Floating-Point Family from GF4 to GF1024 with a Lucas-Exact Integer Identity
GoldenFloat introduces a phi-derived rule for setting exponent and fraction widths across floating-point formats from 4 to 1024 bits, backed by open RTL generator, Lucas-exact accumulator, and FPGA implementation.
-
MASQ: Accelerating Masked Diffusion via Stage-Wise Multi-Precision Quantization
MASQ claims up to 16.06x speedup and 4.18x energy gain over A100 for masked diffusion via stage-wise multi-precision quantization and specialized hardware units while preserving quality.
-
A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of Large Language Models
SOP post-training quantization for LLMs reports lower weight reconstruction error than per-layer FP8 at 1.5 bpw lower cost using per-layer codebook search and hardware-aware formats.
-
HiFloat4 Format for Language Model Pre-training on Ascend NPUs
HiFloat4 FP4 with stabilization techniques trains dense and MoE language models on Ascend NPUs at relative error within 1% of full-precision baselines.
-
An 83-Format Numeric Catalog with Bit-Exact Conformance Vectors: A Vendor-Neutral Reference for FP8, BF16, MXFP4, and Microscaling Formats
An 83-format numeric catalog with bit-exact conformance vectors and IEEE P3109 cross-walk serving as a vendor-neutral reference for FP8, BF16, MXFP4, and microscaling formats.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.