Pith. sign in

REVIEW 2 major objections 2 minor 286 cited by

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

T0 review · 2 major / 2 minor · reviewed 2026-05-13 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Batch Normalization normalizes each layer's inputs using mini-batch statistics, allowing higher learning rates and faster convergence in deep networks.

desk verdict Batch Norm is a practical architectural tweak that speeds up deep net training with higher learning rates and delivers measurable ImageNet gains, even if the internal covariate shift story is not directly measured. read the letter →

arxiv 1502.03167 v3 pith:V4AFOVJK submitted 2015-02-11 cs.LG

Sergey Ioffe , Christian Szegedy This is my paper · ORCID
classification cs.LG
keywords batchnormalizationinternalcovariateshiftdeepneuralnetworkstrainingaccelerationmini-batchstatisticsimageclassificationregularizationlearningrate
checked against Cost.FunctionalEquationFoundation.LawOfExistence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep networks train slowly because the input distribution to each layer shifts as parameters in earlier layers change, a problem the authors call internal covariate shift. This forces small learning rates and careful initialization, especially when using saturating nonlinearities. The paper integrates normalization directly into the architecture by computing mean and variance over each training mini-batch for every layer, then applying learned scale and shift parameters. The result is that networks can use much higher learning rates, become less sensitive to initialization, and gain a regularizing effect that sometimes removes the need for dropout. On a state-of-the-art image model this reaches target accuracy after 14 times fewer steps and sets a new record on ImageNet when ensembled.

What carries the argument

Batch Normalization, which subtracts the mini-batch mean and divides by the mini-batch standard deviation for each layer's activations before applying learned scale and shift parameters.

What would settle it

A network trained with batch normalization that still requires low learning rates, careful initialization, or more steps than the baseline to reach the same accuracy would falsify the central claim.

Watch

Extended reading notes

Core claim

Making normalization a part of the model architecture and performing it per mini-batch reduces internal covariate shift, so that the same accuracy is reached with far fewer training steps while using higher learning rates and less careful initialization.

Load-bearing premise

The changing distribution of each layer's inputs is the main cause of slow training, and normalizing per mini-batch will reliably reduce this shift without introducing instabilities or needing extensive extra tuning.

Editorial extensions

If this is right

  • Networks can safely use significantly higher learning rates without divergence.
  • Training requires less careful parameter initialization.
  • The regularizing effect can eliminate the need for dropout in some models.
  • Target accuracy is reached after 14 times fewer training steps on image classification tasks.
  • An ensemble achieves 4.9 percent top-5 error on ImageNet, beating prior published results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same per-batch normalization idea could stabilize training in other sequence or graph models where layer input distributions also drift.
  • Smaller batch sizes may limit the reliability of the estimated statistics, pointing to possible variants that use running averages or different grouping.
  • By reducing sensitivity to initialization, the method could make deep learning more accessible outside specialized labs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Batch Normalization as an architectural component that normalizes each layer's inputs to zero mean and unit variance using per-mini-batch statistics, followed by learnable scale and shift parameters. It claims this mitigates internal covariate shift, enabling substantially higher learning rates, reduced sensitivity to initialization, and a regularizing effect that can replace Dropout. Experiments on MNIST and a state-of-the-art ImageNet model report that the same accuracy is reached with 14 times fewer training steps and that an ensemble improves top-5 validation error to 4.9%.

Significance. If the empirical gains hold under the reported conditions, the work is significant: it supplies a practical, low-overhead technique that has become standard in deep-network training pipelines and directly enabled deeper architectures. The paper supplies explicit algorithmic pseudocode, the full training protocol for the ImageNet model, and reproducible speed-up numbers, all of which strengthen its contribution.

major comments (2)
  1. [§4] §4 (ImageNet experiments): no direct metric of internal covariate shift (mean/variance drift, KL divergence, or Wasserstein distance between successive layer-input distributions) is reported for the baseline versus BN networks. Consequently the central causal claim—that the observed 14-fold reduction in training steps stems from reduced ICS rather than from stochastic regularization or improved loss-landscape conditioning—remains unverified.
  2. [§3.2] §3.2, Eq. (3)–(5): the normalization is performed with mini-batch statistics whose variance is itself stochastic; the manuscript provides no analysis or bound showing that this stochasticity reliably decreases (rather than merely reparameterizes) the covariate shift that the authors define in §2.
minor comments (2)
  1. [Figure 1] Figure 1 caption: the legend does not explicitly state which curves include the BN layers and which are the plain baseline, making the speed-up comparison harder to read at a glance.
  2. [§4.1] §4.1: the MNIST results are reported without error bars or the number of independent runs, even though the absolute accuracy differences are small.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback on our manuscript. We respond to each major comment below, providing clarifications and indicating where revisions can be made.

read point-by-point responses

  1. Referee: [§4] §4 (ImageNet experiments): no direct metric of internal covariate shift (mean/variance drift, KL divergence, or Wasserstein distance between successive layer-input distributions) is reported for the baseline versus BN networks. Consequently the central causal claim—that the observed 14-fold reduction in training steps stems from reduced ICS rather than from stochastic regularization or improved loss-landscape conditioning—remains unverified.

    Authors: We acknowledge that direct metrics of internal covariate shift (e.g., distribution distances) are not reported. The primary evidence remains the empirical training speedups and accuracy gains on MNIST and ImageNet, which are consistent with reduced ICS. Other mechanisms such as regularization may contribute, and we can add a short discussion in revision noting the absence of direct ICS quantification while emphasizing the practical benefits. revision: partial

  2. Referee: [§3.2] §3.2, Eq. (3)–(5): the normalization is performed with mini-batch statistics whose variance is itself stochastic; the manuscript provides no analysis or bound showing that this stochasticity reliably decreases (rather than merely reparameterizes) the covariate shift that the authors define in §2.

    Authors: Mini-batch statistics are stochastic by nature, yet the normalization (combined with learnable scale/shift and population statistics at inference) stabilizes each layer's input distribution. We provide no formal bound or analysis of the stochasticity, as the paper is primarily empirical; the consistent speed and accuracy improvements across models indicate a net reduction in effective covariate shift despite the stochastic estimates. revision: no

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity in derivation chain

full rationale

The paper introduces Batch Normalization as an explicit architectural layer that computes per-mini-batch mean and variance, normalizes activations, and applies learnable scale/shift parameters. Its central claims of faster convergence, higher learning rates, and regularization effects are supported by direct empirical comparisons on external benchmarks (e.g., ImageNet accuracy and training steps) rather than any mathematical reduction of a predicted quantity back to a fitted parameter defined from the same data. No equations equate a claimed improvement to an input by construction, and no load-bearing premise relies on self-citation chains or imported uniqueness theorems. The derivation is therefore self-contained and externally falsifiable.

Assumptions & free parameters 1 free parameters · 1 assumptions · 1 invented entities

The approach rests on the domain assumption that input distribution shifts are the dominant training obstacle and on the introduction of two learnable parameters per layer to restore representational power after normalization.

free parameters (1)
  • gamma and beta
    Learnable scale and shift parameters per feature that are fitted during training to allow the network to recover any desired distribution after normalization.
assumptions (1)
  • domain assumption Changing distributions of layer inputs during training slow convergence and require lower learning rates
    Invoked in the opening paragraph to motivate the need for normalization.
invented entities (1)
  • internal covariate shift
    purpose: To name and frame the phenomenon of changing layer-input distributions as the core training difficulty
    New term introduced to describe the problem the method targets; no independent measurement provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." pith.science (2026). https://pith.science/paper/V4AFOVJK

@misc{pith2026150203167,
  author       = {Pith},
  title        = {Pith review of: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4AFOVJK}},
  note         = {Machine review of arXiv:1502.03167}
}
read the original abstract

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 286 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 24,381 citations worldwide. See all 286 Pith citations

  1. Into the Darkness: Classical and Type II Cepheids in the Zona Galactica Incognita

    astro-ph.SR 2019-08 conditional novelty 8.0 of 10

    A near-infrared census of the hidden far side of the Milky Way disk yields over 1,000 new classical and type II Cepheids, new extinction measurements, and tracers of the warp and age structure of the disk.

  2. Simplex Demixing: Disentangling Multiple Light-Flavor Jets at Colliders

    hep-ph 2026-07 conditional novelty 7.0 of 10

    Simplex demixing recovers T mutually irreducible jet-flavor topics from M mixed samples via the (T−1)-simplex geometry of a multi-category classifier, demonstrated on Pythia dijets.

  3. Determining star formation histories and age-metallicity relations with convolutional neural networks

    astro-ph.GA 2026-05 unverdicted novelty 7.0 of 10

    A CNN with attention and shared latent space recovers SFHs and metallicities from spectro-photometric data with ~0.12 dex age and ~0.03 dex metallicity dispersion while running thousands of times faster than full spec...

  4. Physics-informed, Generative Adversarial Design of Funicular Shells

    cs.CE 2026-04 unverdicted novelty 7.0 of 10

    A modified DCGAN with an auxiliary membrane-factor discriminator generates previously unseen, high-membrane-factor funicular shell geometries for pure-compression design.

  5. Machine-learning techniques for model-independent searches in dijet final states

    hep-ex 2025-12 accept novelty 7.0 of 10

    Five ML anomaly-detection methods enhance model-agnostic dijet searches at CMS, and a weakly supervised tagger identifies hadronic top-quark decays in data nearly as well as a supervised classifier.

  6. Deep Learning for CMB Foreground Removal and Beam Deconvolution: A U-Net GAN Approach

    astro-ph.IM 2025-08 unverdicted novelty 7.0 of 10

    A U-Net GAN reconstructs CMB T and E maps from Planck-like simulations with foregrounds and systematics, achieving under 1% error outside the Galactic region and demonstrating first-time correction for non-circular be...

  7. Beyond Topological Self-Explainable GNNs: A Formal Explainability Perspective

    cs.LG 2025-02 accept novelty 7.0 of 10

    Self-explainable GNNs provably optimize minimal explanations that match prime implicants only for motif-based tasks, and a dual-channel extension recovers better rules.

  8. Histogram-Equalized Quantization for logic-gated Residual Neural Networks

    cs.LG 2025-01 conditional novelty 7.0 of 10

    HEQ sets quantization step sizes from weight quantiles, achieving competitive CIFAR-10 accuracy and enabling logic-gated residual networks on STL-10.

  9. End-to-End Multi-Track Reconstruction using Graph Neural Networks at Belle II

    physics.ins-det 2024-11 conditional novelty 7.0 of 10

    A GNN-based end-to-end track finder for the Belle II drift chamber reconstructs displaced tracks at 85.4% efficiency with a 2.5% fake rate, outperforming the baseline algorithm at 52.2%.

  10. High Fidelity Neural Audio Compression

    eess.AS 2022-10 accept novelty 7.0 of 10

    EnCodec is an end-to-end trained streaming neural audio codec that uses a single multiscale spectrogram discriminator and a gradient-normalizing loss balancer to achieve higher fidelity than prior methods at the same ...

  11. A Simple Framework for Contrastive Learning of Visual Representations

    cs.LG 2020-02 accept novelty 7.0 of 10

    SimCLR learns visual representations by contrasting augmented views of the same image and reaches 76.5% ImageNet top-1 accuracy with a linear classifier, matching a supervised ResNet-50.

  12. On the Acceleration of Deep Learning Model Parallelism with Staleness

    cs.LG 2019-09 reject novelty 7.0 of 10

    DSP decouples forward and backward passes in model-parallel deep CNN training by giving each layer block a preset staleness, yielding speedups up to 4.8x with comparable or better accuracy.

  13. DeepIST: Deep Image-based Spatio-Temporal Network for Travel Time Estimation

    cs.CV 2019-09 conditional novelty 7.0 of 10

    DeepIST represents a route as a sequence of multi-channel images and combines a line-focused 2D CNN with a 1D CNN to estimate travel time, reporting 24-25% lower MAE than prior models on two taxi datasets.

  14. Riemannian batch normalization for SPD neural networks

    cs.LG 2019-09 conditional novelty 7.0 of 10

    A Riemannian batch normalization layer for SPD neural networks, using barycenter centering and learned bias, improves classification accuracy and robustness to scarce data.

  15. AutoGAN: Neural Architecture Search for Generative Adversarial Networks

    cs.CV 2019-08 conditional novelty 7.0 of 10

    AutoGAN applies reinforcement-learning-based neural architecture search to GAN generators, discovering a CIFAR-10 architecture with FID 12.42 and an STL-10 FID 31.01, both state of the art in 2019.

  16. MVB: A Large-Scale Dataset for Baggage Re-Identification and Merged Siamese Networks

    cs.CV 2019-07 unverdicted novelty 7.0 of 10

    Releases MVB, a multi-view baggage re-identification dataset with 4519 identities and 22660 images, plus a merged Siamese network baseline evaluated on it.

  17. Learning to learn with quantum neural networks via classical neural networks

    quant-ph 2019-07 unverdicted novelty 7.0 of 10

    Classical RNNs trained on small instances provide parameter initializations for QAOA and VQE that reduce total optimization iterations and generalize across problem sizes.

  18. IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery

    physics.comp-ph 2019-07 unverdicted novelty 7.0 of 10

    IRNet uses per-layer residual shortcuts in fully connected networks to achieve better prediction accuracy and training convergence than prior ML methods on OQMD and Materials Project datasets for material properties.

  19. Importance Estimation for Neural Network Pruning

    cs.LG 2019-06 unverdicted novelty 7.0 of 10

    Taylor-expansion importance scoring enables layer-agnostic pruning of neural networks that outperforms prior methods on ImageNet accuracy-FLOPs trade-offs.

  20. A Geometry-Limited Identification Floor and Its Consequences for Voice-Clone Attribution in Professional Voice Actors

    eess.AS 2026-07 conditional novelty 6.0 of 10

    On 1,168 professional voice actors, a misidentification floor in speaker embeddings survives calibration, normalization, and discriminative re-ranking, and the same floor makes fixed-threshold voice-clone attribution ...

  21. Understanding electricity consumption behaviour through Inverse Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    AIRL-recovered reward functions show that the 2022 energy crisis and heatwave reshaped Italian households' cooling responses heterogeneously, sometimes durably, with time-of-use as an independent dimension.

  22. Higher-order effects in amplitude-assisted polarisation extraction with machine-learning techniques

    hep-ph 2026-07 unverdicted novelty 6.0 of 10

    First NLO-QCD amplitude-assisted ML regression for longitudinal-boson production rate in di-boson events at the LHC, benchmarked against random forests.

  23. A Dual-domain Refinement Network with FBP-based Jacobian Learning for Sparse-view Dual-Energy CT Material Decomposition

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    DECT-DRNet combines an FBP-based learnable Jacobian approximation with dual-domain Fourier regularization to improve accuracy of multi-material decomposition from sparse-view dual-energy CT data.

  24. Acceleration of an algebraic multigrid pressure solver using graph neural networks

    physics.comp-ph 2026-06 unverdicted novelty 6.0 of 10

    A modified graph convolutional isomorphism network predicts polynomial coefficients for a sparse pseudo-inverse AMG smoother, cutting V-cycles and delivering 4-37% wall-clock speedups while generalizing to larger and ...

  25. IV-Net: A neural network for elliptic PDEs with random and highly varying coefficients

    math.NA 2026-05 unverdicted novelty 6.0 of 10

    IV-Net is a multigrid-inspired convolutional neural operator that approximates solutions to linear elliptic PDEs with high-contrast coefficients and shows better accuracy than POD and other neural operators on heterog...

  26. CogAdapt: Transferring Clinical ECG Foundation Models to Wearable Cognitive Load Assessment via Lead Adaptation

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    CogAdapt adapts clinical ECG foundation models to 3-lead wearable signals for cognitive load assessment via a LeadBridge adapter and ProFine progressive fine-tuning, outperforming scratch-trained models with macro-F1 ...

  27. Q-PhotoNAS: Hybrid Quantum Neural Architecture Search Framework on Photonic Devices

    quant-ph 2026-05 unverdicted novelty 6.0 of 10

    Q-PhotoNAS applies genetic algorithm search to jointly optimize classical preprocessing, phase encoding, and photonic circuit structure for hybrid quantum-classical models, reporting 99.44% and 98.78% accuracy on Digi...

  28. A Dual Physics-Informed Kolmogorov-Arnold Neural Network Framework for Continuum Topology Optimization

    cs.CE 2026-05 unverdicted novelty 6.0 of 10

    Dual HRKAN framework (DPIKAN-TO) for topology optimization with one network predicting displacements and another handling sensitivity-based design updates.

  29. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    LeWM is the first end-to-end trainable JEPA from pixels that uses only two loss terms for stable training and fast planning on 2D/3D control tasks.

  30. Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    TaperNorm gradually removes internal normalization in pre-norm transformers via learned gates that reach zero, revealing final norm as a scale anchor and enabling up to 1.18x faster KV-cached decoding with small loss ...

  31. MiniFool -- Physics-Constraint-Aware Minimizer-Based Adversarial Attacks in Deep Neural Networks

    cs.LG 2025-11 conditional novelty 6.0 of 10

    MiniFool generates uncertainty-aware adversarial perturbations and uses the attack strength needed to flip a classification as a measure of how robust the network's decision is.

  32. Deep Learning for High Speed Optical Coherence Elastography with a Fiber Scanning Endoscope

    eess.SP 2025-09 conditional novelty 6.0 of 10

    A miniaturized fiber-scanning endoscope paired with a spatio-temporal DenseNet estimates localized tissue elasticity from sparse wave-field images, with lower phantom errors than conventional phase-tracking elastography.

  33. VISP: Volatility Informed Stochastic Projection for Adaptive Regularization

    cs.LG 2025-09 reject novelty 6.0 of 10

    VISP applies gradient-volatility-scaled stochastic projection to activations and reports improved test error on three image benchmarks, but without error bars or code.

  34. TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings

    cs.SE 2025-08 conditional novelty 6.0 of 10

    TriagerX combines dual-transformer content rankings with developer interaction history to improve top-k accuracy for developer and component recommendations in bug triaging across five datasets.

  35. Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Introduces layer-wise learning signals combining knowledge distillation and local errors into Equilibrium Propagation, enabling scalable training of deep VGG-style CRNNs with SOTA results on CIFAR-10 and CIFAR-100.

  36. Anomaly Detection for IoT Global Connectivity

    cs.NI 2025-08 conditional novelty 6.0 of 10

    An unsupervised roaming-signaling pipeline flags IoT fleet connectivity issues, but the headline results are weakened by training/test overlap.

  37. Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks

    cs.RO 2025-07 conditional novelty 6.0 of 10

    The paper introduces MTBench, a GPU-accelerated benchmark for massively parallel multi-task RL, and reports experiments suggesting on-policy methods outperform off-policy baselines while value learning limits MTRL per...

  38. Perspective from a Higher Dimension: Can 3D Geometric Priors Help Visual Floorplan Localization?

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Self-supervised 3D geometric priors, trained with multi-view and view-scene contrastive constraints, transferred into an F3Loc-style observation model improve visual floorplan localization accuracy on Structured3D and...

  39. Enhancing Fatigue Detection through Heterogeneous Multi-Source Data Integration and Cross-Domain Modality Imputation

    cs.RO 2025-07 conditional novelty 6.0 of 10

    A framework that imputes missing high-fidelity sensor channels (EEG, ECG) from shared wearable signals improves fatigue detection accuracy in a sensor-constrained target dataset in several configurations.

  40. Interpretable Prediction of Lymph Node Metastasis in Rectal Cancer MRI Using Variational Autoencoders

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A VAE-based multiple instance learning model predicts rectal cancer lymph node metastasis from MRI with cross-validated AUC 0.858 on 168 patients, but the state-of-the-art and radiologist-exceeding claims are weakly s...

  41. Emulating CO Line Radiative Transfer with Deep Learning

    astro-ph.IM 2025-07 conditional novelty 6.0 of 10

    COEmuNet, a 3D CNN, reproduces Pomme's CO J=2-1 line images of binary-perturbed AGB winds to ~7% median relative error while running ~1000x faster on a GPU.

  42. Electricity Market Predictability: Virtues of Machine Learning and Links to the Macroeconomy

    econ.GN 2025-07 reject novelty 6.0 of 10

    Machine learning models, especially boosted trees and GLM, predict Singapore daily electricity returns out-of-sample, and a correlation-penalized ensemble beats all individual models, with predictability concentrated ...

  43. Energy-Efficient Supervised Learning with a Binary Stochastic Forward-Forward Algorithm

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Binary stochastic forward-forward training reaches near-real-valued forward-forward accuracy on image benchmarks while estimating 10-100x energy savings in p-bit hardware.

  44. Hardware-software co-exploration with racetrack memory based in-memory computing for CNN inference in embedded systems

    cs.ET 2025-07 conditional novelty 6.0 of 10

    A simulated racetrack-memory in-memory CNN accelerator using write-to-shift conversion and shift-based multipliers achieves up to 83.5x energy-efficiency improvement and 1.68x performance gain over an unoptimized 8-bi...

  45. Neural Augmented Kalman Filters for Road Network assisted GNSS positioning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A TGNN selects road segments and predicts road-measurement uncertainty inside a Kalman filter, reducing urban GNSS horizontal error at the 95th percentile by 29% relative to GNSS-only.

  46. AG-VPReID 2025: Aerial-Ground Video-based Person Re-identification Challenge Results

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new challenge and dataset for high-altitude aerial-ground video person re-identification, with the winning team achieving 72.28% Rank-1 in the aerial-to-ground task.

  47. EFormer: An Effective Edge-based Transformer for Vehicle Routing Problems

    cs.LG 2025-06 conditional novelty 6.0 of 10

    EFormer, an edge-input transformer with a mixed-score precoder and parallel graph and node encoders, improves TSP and CVRP optimality gaps over prior edge-based neural heuristics.

  48. Domain Generalization for Person Re-identification: A Survey Towards Domain-Agnostic Person Matching

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A systematic survey that categorizes domain-generalizable person re-identification methods and compares their cross-domain performance.

  49. Vector Representations of Vessel Trees

    eess.IV 2025-06 conditional novelty 6.0 of 10

    VeTTA encodes a vascular tree into one vector and recursively decodes it into a geometrically accurate, topologically valid tree, outperforming voxel-based autoencoders on reconstruction metrics.

  50. NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationarity

    cs.LG 2025-06 conditional novelty 6.0 of 10

    NILMFormer, a Transformer that normalizes each smart-meter window and feeds the window statistics back into the model, improves appliance-level power disaggregation accuracy over prior NILM models on four datasets.

  51. Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data

    cs.AR 2025-05 conditional novelty 6.0 of 10

    Chameleon is a 40-nm CMOS accelerator that performs end-to-end few-shot and continual learning on-chip using prototypical networks and TCN embeddings, and runs keyword spotting at 3.1 uW.

  52. CHIP: Chameleon Hash-based Irreversible Passport for Robust Deep Model Ownership Verification and Active Usage Control

    cs.CR 2025-05 conditional novelty 6.0 of 10

    CHIP hides a chameleon-hash signature in a neural network's normalization layers, enabling ownership verification, per-user access control, and user tracing without retraining.

  53. Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Hybrid Batch Normalisation improves federated learning accuracy by combining local batch statistics with global statistics, and derives those global statistics from the pre-update global model.

  54. Virtual Dosimetrists: A Radiotherapy Training "Flight Simulator"

    physics.med-ph 2025-05 conditional novelty 6.0 of 10

    A dose-prediction network conditioned on CLIP text embeddings can generate and iteratively revise radiotherapy dose distributions from simple language prompts, creating a fast training simulator for plan quality review.

  55. Tagging fully hadronic exotic decays of the vectorlike $\mathbf{B}$ quark using a graph neural network

    hep-ph 2025-05 conditional novelty 6.0 of 10

    A GNN-based search could give the HL-LHC an exclusion reach near 2.4 TeV for vectorlike B quarks decaying fully hadronically through b plus a singlet scalar, with performance comparable to semileptonic searches.

  56. Document Image Rectification Bases on Self-Adaptive Multitask Fusion

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A multitask fusion network with inter-task feature aggregation and gating reports state-of-the-art document dewarping results on DIR300, DocUNet, and DocReal, subject to comparison caveats.

  57. DeepSync: A Learning Framework for Pervasive Localization using Code Synchronization on Compressed Cellular Spectrum

    eess.SY 2025-05 reject novelty 6.0 of 10

    DeepSync claims meter-level localization from nonlinearly compressed LTE spectrum using a cross-attention network, but its real-world test applies the nonlinear transformation in software rather than using actual low-...

  58. Boosting Statistic Learning with Synthetic Data from Pretrained Large Models

    stat.ML 2025-05 reject novelty 6.0 of 10

    The paper claims synthetic tabular data generated by pass-through Stable Diffusion, filtered by Wasserstein distance or hypothesis tests, improves predictive accuracy, but the evidence is weakened by missing baselines...

  59. Development and evaluation of a deep learning algorithm for German word recognition from lip movements

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A CNN-GRU lip-reading model reaches 87% accuracy on known speakers and 63% on unseen speakers for 18 German words, the first such system for German.

  60. Forget the Data and Fine-Tuning! Just Fold the Network to Compress

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Model folding compresses a network by k-means clustering similar neurons across adjacent layers and repairing activation statistics without data (Fold-AR, Fold-DIR), surpassing prior data-free methods at high sparsity.

See all 286 Pith citations

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · cited by 286 Pith papers (see all)

  1. [1]

    Understanding the difficulty of training deep feedforward neural networks

    Bengio, Yoshua and Glorot, Xavier. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS 2010, volume 9, pp.\ 249--256, May 2010

    work page 2010

  2. [2]

    Large scale distributed deep networks

    Dean, Jeffrey, Corrado, Greg S., Monga, Rajat, Chen, Kai, Devin, Matthieu, Le, Quoc V., Mao, Mark Z., Ranzato, Marc'Aurelio, Senior, Andrew, Tucker, Paul, Yang, Ke, and Ng, Andrew Y. Large scale distributed deep networks. In NIPS, 2012

    work page 2012

  3. [3]

    Natural neural networks

    Desjardins, Guillaume and Kavukcuoglu, Koray. Natural neural networks. (unpublished)

    work page

  4. [4]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12: 0 2121--2159, July 2011. ISSN 1532-4435

    work page 2011

  5. [5]

    Knowledge matters: Importance of prior information for optimization

    G \" u l c ehre, C aglar and Bengio, Yoshua. Knowledge matters: Importance of prior information for optimization. CoRR, abs/1301.4083, 2013

    work page Pith review arXiv 2013

  6. [6]

    Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

    He , K., Zhang , X., Ren , S., and Sun , J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification . ArXiv e-prints, February 2015

    work page 2015

  7. [7]

    and Oja, E

    Hyv\" a rinen, A. and Oja, E. Independent component analysis: Algorithms and applications. Neural Netw., 13 0 (4-5): 0 411--430, May 2000

    work page 2000

  8. [8]

    A literature survey on domain adaptation of statistical classifiers, 2008

    Jiang, Jing. A literature survey on domain adaptation of statistical classifiers, 2008

    work page 2008

Show all 24 references
  1. [9]

    Gradient-based learning applied to document recognition

    LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, November 1998 a

    1998

  2. [10]

    Efficient backprop

    LeCun, Y., Bottou, L., Orr, G., and Muller, K. Efficient backprop. In Orr, G. and K., Muller (eds.), Neural Networks: Tricks of the trade. Springer, 1998 b

    1998

  3. [11]

    Nonlinear image representation using divisive normalization

    Lyu, S and Simoncelli, E P. Nonlinear image representation using divisive normalization. In Proc. Computer Vision and Pattern Recognition, pp.\ 1--8. IEEE Computer Society, Jun 23-28 2008. doi:10.1109/CVPR.2008.4587821

    2008 doi

  4. [12]

    Rectified linear units improve restricted boltzmann machines

    Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In ICML, pp.\ 807--814. Omnipress, 2010

    2010

  5. [13]

    On the difficulty of training recurrent neural networks

    Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013 , pp.\ 1310--1318, 2013

    2013

  6. [14]

    Parallel training of deep neural networks with natural gradient and parameter averaging

    Povey, Daniel, Zhang, Xiaohui, and Khudanpur, Sanjeev. Parallel training of deep neural networks with natural gradient and parameter averaging. CoRR, abs/1410.7455, 2014

    2014

  7. [15]

    Deep learning made easier by linear transformations in perceptrons

    Raiko, Tapani, Valpola, Harri, and LeCun, Yann. Deep learning made easier by linear transformations in perceptrons. In International Conference on Artificial Intelligence and Statistics ( AISTATS ) , pp.\ 924--932, 2012

    2012

  8. [16]

    ImageNet Large Scale Visual Recognition Challenge , 2014

    Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. ImageNet Large Scale Visual Recognition Challenge , 2014

    2014

  9. [17]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Saxe, Andrew M., McClelland, James L., and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. CoRR, abs/1312.6120, 2013

    2013

  10. [18]

    Improving predictive inference under covariate shift by weighting the log-likelihood function

    Shimodaira, Hidetoshi. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90 0 (2): 0 227--244, October 2000

    2000

  11. [19]

    Dropout: A simple way to prevent neural networks from overfitting

    Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15 0 (1): 0 1929--1958, January 2014

    1929

  12. [20]

    On the importance of initialization and momentum in deep learning

    Sutskever, Ilya, Martens, James, Dahl, George E., and Hinton, Geoffrey E. On the importance of initialization and momentum in deep learning. In ICML (3), volume 28 of JMLR Proceedings, pp.\ 1139--1147. JMLR.org, 2013

    2013

  13. [21]

    Going deeper with convolutions

    Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. CoRR, abs/1409.4842, 2014

    2014

  14. [22]

    A convergence analysis of log-linear training

    Wiesler, Simon and Ney, Hermann. A convergence analysis of log-linear training. In Shawe-Taylor, J., Zemel, R.S., Bartlett, P., Pereira, F.C.N., and Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 24, pp.\ 657--665, Granada, Spain, December 2011

    2011

  15. [23]

    Mean-normalized stochastic gradient for large-scale deep learning

    Wiesler, Simon, Richard, Alexander, Schl \"u ter, Ralf, and Ney, Hermann. Mean-normalized stochastic gradient for large-scale deep learning. In IEEE International Conference on Acoustics, Speech, and Signal Processing, pp.\ 180--184, Florence, Italy, May 2014

    2014

  16. [24]

    Deep image: Scaling up image recognition, 2015

    Wu, Ren, Yan, Shengen, Shan, Yi, Dang, Qingqing, and Sun, Gang. Deep image: Scaling up image recognition, 2015

    2015

Pith tools

Reviewed May 13, 2026 · model on record in the stance chip above.