Pith. sign in

REVIEW 3 major objections 4 minor 48 references

SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Deformable image registration improves when both feature extraction and deformation prediction are routed per location and per direction, via a mixture of attention heads in the encoder and a spatial heterogeneous mixture of experts in the

desk verdict Solid empirical MoE-for-registration paper, but the SHMoE ablation doesn't isolate routing from added capacity. read the letter →

arxiv 2509.20073 v2 pith:TW2EBFI6 submitted 2025-09-24 cs.CV

Yuxi Zheng , Jianhui Feng , Tianran Li , Marius Staring , Yuchuan Qiao This is my paper
classification cs.CV MSC 68T0768U1092C55
keywords deformableimageregistrationmixtureofexpertsattentionheadsencoder-decoderdeformationfield3DmedicalimagingroutingclassificationlossBTCV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that deformable image registration improves when feature extraction and deformation prediction are made heterogeneous and specialized. It introduces SHMoAReg, which uses a mixture of attention heads in the encoder so each image token can select its own attention heads, and a spatial heterogeneous mixture of experts in the decoder so each voxel can select direction-specific convolution kernels (sizes 1, 3, 5) for predicting deformation. On the BTCV abdominal CT dataset without pre-affine alignment, the method reports 65.58% Dice and 2.46 mm ASSD, beating the second-best method by 5.0% Dice and 0.35 mm, with similar trends on brain MR. Ablations attribute 6.42% Dice to the decoder SHMoE and 1.94% to the encoder MoA. A diffeomorphic variant shows the gains are not bought by folding.

What carries the argument

Two routed mixture modules. MoA: for each query token, a linear router picks top-4 of 12 attention-head experts (shared keys/values, per-expert query/output projections); the output is a weighted sum of selected experts. SHMoE: a learnable routing tensor T(f) with top-1 nonzero entry per voxel selects one of three convolution experts (kernel sizes 1, 3, 5) to produce the residual deformation component in a given direction; the routing tensor is trained with a binary cross-entropy loss whose labels come from thresholding the gradient magnitude of the MSE similarity loss. The routing losses are what force specialization rather than just adding capacity.

What would settle it

Keep the SHMoE expert set (kernel sizes 1, 3, 5) but replace the learned routing tensor with random per-voxel expert selection or with a fixed uniform mixture, retraining under identical settings on BTCV; if Dice stays near 65.58%, the routing classification loss and learned per-voxel specialization are not the cause of the improvement. A second check: train on isotropic OASIS and on a deliberately anisotropic resampled version of the same data; if the SHMoE advantage does not grow with anisotropy, the per-direction specialization story is weakened.

Watch

Extended reading notes

Core claim

The central discovery is that applying mixture-of-experts at both ends of an encoder–decoder registration network, with routing at token and voxel granularity, yields consistent and substantial accuracy gains over homogeneous CNN/Transformer baselines. The encoder's Mixture of Attention heads lets each query token choose its own subset of attention projections instead of using all heads uniformly. The decoder's Spatial Heterogeneous Mixture of Experts predicts deformation per direction (x/y/z) per voxel, with experts realized as convolutions of kernel sizes 1, 3, and 5, so a voxel can choose a small-, medium-, or large-receptive-field deformation estimate. Training uses a routing classificat

Load-bearing premise

The routing labels come from the gradient magnitude of the same MSE similarity loss that trains the network, so the claim that experts specialize depends on this gradient signal correctly identifying which expert is wrong at each voxel; if this signal is uninformative, the reported gains could come from added convolution capacity rather than from per-voxel heterogeneous routing.

Editorial extensions

If this is right

  • If correct, registration on anisotropic data (e.g. CT with 3×3×2 mm spacing) can be improved by letting each direction of the deformation field use a different receptive-field scale.
  • The 6.42% Dice drop when SHMoE is replaced by a single 3×3×3 convolution implies that per-voxel heterogeneous routing, not just multi-kernel capacity, drives the improvement.
  • Adding SHMoE only at full and half resolution is best; lower-resolution routing hurts, suggesting a design rule for where to put voxel-level experts in pyramid decoders.
  • The same routing-with-gradient-labels recipe may transfer to other dense prediction tasks (segmentation, motion estimation) where per-location specialization is desirable.
  • The diffeomorphic variant shows the accuracy gains are achievable without increased folding, decoupling the mixture mechanism from topology issues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gradient-magnitude-based expert labels are a form of online credit assignment that may be unstable early in training; a falsifiable test is to compare against random expert labels while keeping the same expert set and capacity.
  • The paper's interpretability analysis (experts specialize across resolutions and spatial locations) suggests MoE could serve as an inspection tool for understanding which features and which receptive fields drive registration, but the authors do not connect expert selection to anatomical semantics.
  • Because the improvement is largest on anisotropic CT, one testable extension is to vary voxel spacing systematically and check whether SHMoE's benefit scales with anisotropy; if not, the direction-heterogeneous story would need revision.
  • The routing classification loss uses only the similarity gradient, ignoring the regularizer; adding the regularizer's gradient to the label signal might change which experts are selected and could be a cheap improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. SHMoAReg introduces mixture-of-experts modules into an encoder-decoder 3D deformable registration network. The encoder replaces standard Swin Transformer attention heads with Mixture of Attention Heads (MoA), selecting a per-token top-k subset of attention experts. The decoder employs Spatial Heterogeneous Mixture of Experts (SHMoE) at full and half resolutions, where each voxel is routed to one of three convolution experts with kernel sizes 1, 3, and 5 to predict the deformation in each direction. Training combines MSE similarity, smoothness regularization, and a routing classification loss whose labels are derived from gradient magnitudes of the similarity loss. Evaluated on OASIS (brain MR) and BTCV (abdominal CT without pre-affine), the method reports Dice 79.95% on OASIS and 65.58% on BTCV, with ASSD 0.31 mm and 2.46 mm respectively, plus a diffeomorphic variant. Ablations on BTCV attribute 1.94% Dice to MoA and 6.42% Dice to SHMoE, and the paper includes expert-load analyses across resolutions and voxels.

Significance. If the reported gains are robust, the paper makes a useful contribution: it is the first to apply MoE-style routing to deformable image registration, it reports consistent improvements over six existing methods on two datasets, includes a diffeomorphic variant, and provides an interpretability analysis of expert loads. The experiments are comprehensive in scope and the architecture is clearly specified. However, the central mechanistic claim—that per-voxel, per-direction routing is the active ingredient behind the largest Dice gain—is currently undermined by a confounded ablation: the SHMoE module differs from its baseline in kernel diversity, parameter count, and routing mechanism simultaneously. The routing labels also come from the same similarity loss used for training, making the supervision signal self-referential. The interpretability analysis, while attractive, is mostly descriptive. These issues are fixable with targeted control experiments, but they are load-bearing for the paper's novelty and should be addressed before publication.

major comments (3)
  1. [V-C, Table II] The ablation 'removing only SHMoE' compares SHMoE (three expert convolutions with kernels 1/3/5, a learned routing tensor T(f), and the L_rc auxiliary loss) against a single three-channel convolution of kernel size 3. This changes at least three factors simultaneously: kernel diversity, parameter count/FLOPs, and the presence of a routing mechanism. The 6.42-point Dice gain cannot therefore be uniquely attributed to per-voxel routing. Please add a non-routed control with comparable multi-kernel capacity (e.g., a sum of parallel kernels 1/3/5 convolutions, or the same expert set with fixed/random routing assignments), and report the parameter count and FLOPs for each ablation row. A similar caveat applies to the MoA ablation. Without such a control, the manuscript's central mechanistic claim is not uniquely supported.
  2. [III-C, Eqs. (7)-(8)] The routing labels Y are built from the gradient magnitude of L_sim, which is the same similarity loss used to train the registration network. The implicit axiom is that a large gradient magnitude at a voxel indicates that the currently selected expert is wrong. This is non-obvious and is exactly the assumption that needs validation; if it fails, L_rc becomes an uninformative auxiliary loss and the observed SHMoE gain could stem from added capacity or training dynamics rather than routed specialization. Please provide evidence for this assumption: e.g., compare L_rc trained with random labels, or with labels from a different loss (smoothness), or show that changing the routed expert reduces L_sim conditioned on expert capacity. Reporting routing-label accuracy on a held-out set would also help.
  3. [V-D, Figs. 4-6] The interpretability analysis currently shows that decoder expert loads are almost uniformly distributed and that the per-voxel expert map 'resembles noise.' The paper interprets this as evidence for heterogeneous residual deformation learning, but a random-looking gate is also compatible with the gate learning no meaningful spatial structure. To support the specialization claim, please quantify whether the selected expert correlates with local anatomical or deformation properties (e.g., displacement magnitude, Jacobian determinant, organ boundary location) and compare the observed load distribution with a chance baseline. This would make the interpretability contribution more than descriptive.
minor comments (4)
  1. [Eqs. (1)-(2)] The notation in Eq. (1) is ambiguous: (W^att_i V W_v) · W^o_i should be W^att_i (V W_v) W^o_i. Parentheses would help avoid confusion.
  2. [Abstract and Table I] The abstract says 'notable increase from 60.58% to 65.58% in Dice score'; please clarify in the text that 60.58% is the second-best comparison method (2casVM), not the authors' own baseline, to avoid misreading.
  3. [V-B, Fig. 4] The statement that the expert ID map 'appears sufficiently fine-grained to resemble noise' undersells the intended interpretability contribution. A quantitative assessment of the map's spatial structure would strengthen the claim.
  4. [Table I caption] The Wilcoxon signed-rank test is mentioned only in the caption. Please specify in the main text how many test pairs were used and whether the test is paired across subjects; this is important for interpreting the asterisk markers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SHMoAReg's claims are supported by external held-out Dice/ASSD evaluation, and neither the self-referential routing-label heuristic nor the single non-load-bearing author self-citation makes any result equivalent by construction.

full rationale

The paper's central claims are empirical: SHMoAReg is compared against external methods on held-out OASIS and BTCV test sets using Dice, ASSD, and Jacobian determinants. No parameter is fitted to the evaluation metric and then reported as a prediction; the MoA and SHMoE modules are trained end-to-end with an unsupervised similarity loss. The routing classification loss L_rc (Eq. 7) uses labels derived from the gradient of L_sim, which is a self-referential auxiliary training signal, but it is not a derivation of the final result: Eq. (7) is a cross-entropy regularizer, not an identity with L_sim, and the reported registration quality is measured externally. The only author self-citation is [13], a prior paper by co-author Qiao cited in a general list of encoder-decoder DIR methods; it is not load-bearing. The ablation replacing SHMoE with a single kernel-3 convolution changes capacity and kernel diversity simultaneously, which is an experimental confound about attribution, not a circularity: the improvement is not equivalent by construction to the ablation. No prediction in the paper reduces to a fitted constant or to a self-citation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The method rests on standard deep-learning supervision plus a self-generated routing label. The main external check is test-set Dice and ASSD, not an independent mechanistic validation of routing. Hyperparameters and module choices (lambda, q, top-k, kernel sizes) are fitted to validation or chosen by hand, and the routing supervision is generated from the same objective the network optimizes.

free parameters (6)
  • lambda_r = 0.01 (OASIS), 0.1 (BTCV)
    Weight of the deformation regularization term, selected on validation results (Section IV-B).
  • lambda_rc = 0.001
    Weight of the routing classification loss, fixed across datasets (Section IV-B).
  • q = 50%
    Quantile threshold for the gradient-based error signal used to construct routing labels (Section III-C).
  • MoA top-k = 4 of 12
    Number of selected attention experts per token, chosen as part of the architecture and not ablated.
  • SHMoE top-k = 1 of 3
    Number of selected deformation experts per voxel per direction, chosen as part of the architecture.
  • expert kernel sizes = 1, 3, 5
    Hand-picked kernel sizes for the three experts in SHMoE; no ablation over alternative kernel sets is provided.
assumptions (5)
  • domain assumption MSE similarity between warped and fixed images is a valid training signal for anatomical alignment
    The method is unsupervised and uses intensity differences only; no anatomical correspondence labels are used during training (Section IV-B).
  • ad hoc to paper The gradient magnitude of L_sim is a useful indicator of which expert should be selected
    Routing labels are generated from epsilon, the gradient-based error signal, with no independent validation of this proxy (Section III-C, Eq. 7).
  • domain assumption Anisotropic voxel spacing makes per-direction deformation experts beneficial
    BTCV has 3x3x2 mm spacing; the paper motivates SHMoE from directional structure without directly measuring direction-specific variability (Sections I and V).
  • domain assumption The Swin Transformer backbone from TransMorph-small is an adequate feature extractor
    It is used as the shared encoder, so the overall result depends on the quality of this pretrained architecture (Section III-A, IV-B).
  • domain assumption Benchmark labels (35 brain labels, 4 organ labels) are valid for evaluating registration
    Dice and ASSD are computed on these anatomical labels; their correctness is not scrutinized in the paper (Section IV-D).
invented entities (2)
  • Routing tensor T(f)
    purpose: Selects top-k convolution experts per voxel in SHMoE layers to generate direction-specific deformation fields
    Its effectiveness is evaluated only through end-to-end Dice and ASSD in the same paper; there is no separate measurement of routing quality.
  • Expert labels Y constructed from gradient-based error signal
    purpose: Supervises the routing tensor with binary cross-entropy
    The labels are derived from the same loss gradient used for registration, so they do not provide external evidence for the routing mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads." pith.science (2026). https://pith.science/paper/TW2EBFI6

@misc{pith2026250920073,
  author       = {Pith},
  title        = {Pith review of: SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TW2EBFI6}},
  note         = {Machine review of arXiv:2509.20073}
}
read the original abstract

Encoder-Decoder architectures are widely used in deep learning-based Deformable Image Registration (DIR), where the encoder extracts multi-scale features and the decoder predicts deformation fields by recovering spatial locations. However, current methods lack specialized extraction of features (that are useful for registration) and predict deformation jointly and homogeneously in all three directions. In this paper, we propose a novel expert-guided DIR network with Mixture of Experts (MoE) mechanism applied in both encoder and decoder, named SHMoAReg. Specifically, we incorporate Mixture of Attention heads (MoA) into encoder layers, while Spatial Heterogeneous Mixture of Experts (SHMoE) into the decoder layers. The MoA enhances the specialization of feature extraction by dynamically selecting the optimal combination of attention heads for each image token. Meanwhile, the SHMoE predicts deformation fields heterogeneously in three directions for each voxel using experts with varying kernel sizes. Extensive experiments conducted on two publicly available datasets show consistent improvements over various methods, with a notable increase from 60.58% to 65.58% in Dice score for the abdominal CT dataset. Furthermore, SHMoAReg enhances model interpretability by differentiating experts' utilities across/within different resolution layers. To the best of our knowledge, we are the first to introduce MoE mechanism into DIR tasks.

Figures

Figures reproduced from arXiv: 2509.20073 by the authors.

Figure 1
Figure 1. Overview of SHMoAReg. The encoder has two parameter-sharing Swin Transformer backbones with Mixture of Attention heads (MoA) layers. The decoder employs the classic feature pyramid structure, where Spatial Heterogeneous Mixture of Experts (SHMoE) layers are introduced in generating the residual deformation fields at full and 1/2 resolutions [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Details of Mixture of Attention Heads (MoA) layer in the SwinTransformer Block. MoA consists of two mixtures of experts: one for query projection [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the Spatial Heterogeneous Mixture of Experts (SHMoE) layer and training process with routing classification loss [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of the warped images produced by comparison methods and our SHMoAReg (Dice scores shown in the bottom left). For this slice in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Experts’ load percentages for different resolution layers of the (a) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Experts’ load percentages for (a) five image tokens at various locations in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 9 linked inside Pith

  1. [1]

    V oxelMorph: a learning framework for de- formable medical image registration,

    G. Balakrishnanet al., “V oxelMorph: a learning framework for de- formable medical image registration,”IEEE Transactions on Medical Imaging, vol. 38, no. 8, pp. 1788–1800, 2019

    2019

  2. [2]

    AMNet: Adaptive multi-level network for deformable registration of 3D brain MR images,

    T. Cheet al., “AMNet: Adaptive multi-level network for deformable registration of 3D brain MR images,”Medical Image Analysis, vol. 85, p. 102740, 2023

    2023

  3. [3]

    TransMorph: Transformer for unsupervised medical image registration,

    J. Chenet al., “TransMorph: Transformer for unsupervised medical image registration,”Medical Image Analysis, vol. 82, p. 102615, 2022

    2022

  4. [4]

    GroupMorph: medical image registration via grouping network with contextual fusion,

    Z. Tanet al., “GroupMorph: medical image registration via grouping network with contextual fusion,”IEEE Transactions on Medical Imag- ing, 2024

    2024

  5. [5]

    TransMatch: a transformer-based multilevel dual- stream feature matching network for unsupervised deformable image registration,

    Z. Chenet al., “TransMatch: a transformer-based multilevel dual- stream feature matching network for unsupervised deformable image registration,”IEEE Transactions on Medical Imaging, vol. 43, no. 1, pp. 15–27, 2023

    2023

  6. [6]

    Dual-stream pyramid registration network,

    M. Kanget al., “Dual-stream pyramid registration network,”Medical Image Analysis, vol. 78, p. 102379, 2022

    2022

  7. [7]

    Recursive deformable pyramid network for unsu- pervised medical image registration,

    H. Wanget al., “Recursive deformable pyramid network for unsu- pervised medical image registration,”IEEE Transactions on Medical Imaging, vol. 43, no. 6, pp. 2229–2240, 2024

    2024

  8. [8]

    Xmorpher: Full transformer for deformable medical image registration via cross attention,

    J. Shiet al., “Xmorpher: Full transformer for deformable medical image registration via cross attention,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 217–226

    2022

Show all 48 references
  1. [9]

    Large deformation diffeomorphic im- age registration with laplacian pyramid networks,

    T. C. Mok and A. C. Chung, “Large deformation diffeomorphic im- age registration with laplacian pyramid networks,” inMedical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23. ...

    2020

  2. [10]

    Recursive decomposition network for deformable image registration,

    B. Huet al., “Recursive decomposition network for deformable image registration,”IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 10, pp. 5130–5141, 2022

    2022

  3. [11]

    Non-iterative coarse-to-fine transformer networks for joint affine and deformable image registration,

    M. Menget al., “Non-iterative coarse-to-fine transformer networks for joint affine and deformable image registration,” inInternational Confer- ence on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 750–760

    2023

  4. [12]

    ModeT: learning deformable image registration via mo- tion decomposition transformer,

    H. Wanget al., “ModeT: learning deformable image registration via mo- tion decomposition transformer,” inInternational Conference on Med- ical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 740–749

    2023

  5. [13]

    Unsupervised deep learning for FOD-based susceptibility distortion correction in diffusion MRI,

    Y . Qiao and Y . Shi, “Unsupervised deep learning for FOD-based susceptibility distortion correction in diffusion MRI,”IEEE Transactions on Medical Imaging, vol. 41, no. 5, pp. 1165–1175, 2021

    2021

  6. [14]

    Joint progressive and coarse-to-fine registration of brain MRI via deformation field integration and non-rigid feature fusion,

    J. Lvet al., “Joint progressive and coarse-to-fine registration of brain MRI via deformation field integration and non-rigid feature fusion,” IEEE Transactions on Medical Imaging, vol. 41, no. 10, pp. 2788–2802, 2022

    2022

  7. [15]

    Vit-v-net: Vision transformer for unsupervised volumetric medical image registration,

    J. Chenet al., “Vit-v-net: Vision transformer for unsupervised volumetric medical image registration,”Arxiv Preprint Arxiv:2104.06468, 2021

    2021 arXiv

  8. [16]

    Non-iterative coarse-to-fine registration based on single-pass deep cumulative learning,

    M. Menget al., “Non-iterative coarse-to-fine registration based on single-pass deep cumulative learning,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 88–97

    2022

  9. [17]

    How does attention work in vision transformers? A visual analytics attempt,

    Y . Liet al., “How does attention work in vision transformers? A visual analytics attempt,”IEEE Transactions on Visualization and Computer Graphics, vol. 29, no. 6, pp. 2888–2900, 2023

    2023

  10. [18]

    M4oE: A foundation model for medical mul- timodal image segmentation with mixture of experts,

    Y . Jiang and Y . Shen, “M4oE: A foundation model for medical mul- timodal image segmentation with mixture of experts,” inInternational Conference on Medical Image Computing and Computer-Assisted Inter- vention. Springer, 2024, pp. 621–631

    2024

  11. [19]

    Patcher: Patch transformers with mixture of experts for precise medical image segmentation,

    Y . Ouet al., “Patcher: Patch transformers with mixture of experts for precise medical image segmentation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 475–484

    2022

  12. [20]

    A task-conditional mixture-of-experts model for missing modality segmentation,

    P. Novosadet al., “A task-conditional mixture-of-experts model for missing modality segmentation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2024, pp. 34–43

    2024

  13. [21]

    SAM-Med3D-MoE: Towards a non-forgetting segment anything model via mixture of experts for 3D medical image segmen- tation,

    G. Wanget al., “SAM-Med3D-MoE: Towards a non-forgetting segment anything model via mixture of experts for 3D medical image segmen- tation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2024, pp. 552–561

    2024

  14. [22]

    Low-rank mixture-of-experts for continual medical image segmentation,

    Q. Chenet al., “Low-rank mixture-of-experts for continual medical image segmentation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2024, pp. 382–392

    2024

  15. [23]

    Dual-stream pyramid registration network,

    X. Huet al., “Dual-stream pyramid registration network,” inInterna- tional Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2019, pp. 382–390

    2019

  16. [24]

    Adaptive mixtures of local experts,

    R. A. Jacobset al., “Adaptive mixtures of local experts,”Neural Computation, vol. 3, no. 1, pp. 79–87, 1991

    1991

  17. [25]

    Outrageously large neural networks: The sparsely- gated mixture-of-experts layer,

    N. Shazeeret al., “Outrageously large neural networks: The sparsely- gated mixture-of-experts layer,”Arxiv Preprint Arxiv:1701.06538, 2017

    2017 arXiv

  18. [26]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhinet al., “Gshard: Scaling giant models with conditional computation and automatic sharding,”Arxiv Preprint Arxiv:2006.16668, 2020

    2006 arXiv

  19. [27]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Feduset al., “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022

    2022

  20. [28]

    Glam: Efficient scaling of language models with mixture- of-experts,

    N. Duet al., “Glam: Efficient scaling of language models with mixture- of-experts,” inInternational conference on machine learning. PMLR, 2022, pp. 5547–5569

    2022

  21. [29]

    Go wider instead of deeper,

    F. Xueet al., “Go wider instead of deeper,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8779– 8787

    2022

  22. [30]

    Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model,

    A. Liuet al., “Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model,”Arxiv Preprint Arxiv:2405.04434, 2024

    2024 arXiv

  23. [31]

    St-moe: Designing stable and transferable sparse expert models,

    B. Zophet al., “St-moe: Designing stable and transferable sparse expert models,”Arxiv Preprint Arxiv:2202.08906, 2022

    2022 arXiv

  24. [32]

    Openmoe: An early effort on open mixture-of-experts language models,

    F. Xueet al., “Openmoe: An early effort on open mixture-of-experts language models,”Arxiv Preprint Arxiv:2402.01739, 2024

    2024 arXiv

  25. [33]

    Mixture of attention heads: Selecting attention heads per token,

    X. Zhanget al., “Mixture of attention heads: Selecting attention heads per token,”Arxiv Preprint Arxiv:2210.05144, 2022

    2022 arXiv

  26. [34]

    Dense training, sparse inference: Rethinking training of Mixture-of-Experts language models,

    B. Panet al., “Dense training, sparse inference: Rethinking training of Mixture-of-Experts language models,”Corr, 2024

    2024

  27. [35]

    Jetmoe: Reaching llama2 performance with 0.1 m dollars,

    Y . Shenet al., “Jetmoe: Reaching llama2 performance with 0.1 m dollars,”Arxiv Preprint Arxiv:2404.07413, 2024

    2024 arXiv

  28. [36]

    Scaling vision with sparse mixture of experts,

    C. Riquelmeet al., “Scaling vision with sparse mixture of experts,” Advances in Neural Information Processing Systems, vol. 34, pp. 8583– 8595, 2021

    2021

  29. [37]

    Robust mixture-of-expert training for convolutional neural networks,

    Y . Zhanget al., “Robust mixture-of-expert training for convolutional neural networks,” inProceedings of the IEEE/CVF International Con- ference on Computer Vision, 2023, pp. 90–101

    2023

  30. [38]

    Biased mixtures of experts: Enabling computer vision inference under data transfer limitations,

    A. Abbas and Y . Andreopoulos, “Biased mixtures of experts: Enabling computer vision inference under data transfer limitations,”IEEE Trans- actions on Image Processing, vol. 29, pp. 7656–7667, 2020

    2020

  31. [39]

    Network of experts for large-scale image categoriza- tion,

    K. Ahmedet al., “Network of experts for large-scale image categoriza- tion,” inEuropean Conference on Computer Vision. Springer, 2016, pp. 516–532

    2016

  32. [40]

    Hard mixtures of experts for large scale weakly supervised vision,

    S. Grosset al., “Hard mixtures of experts for large scale weakly supervised vision,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6865–6873

    2017

  33. [41]

    Deep mixture of experts via shallow embedding,

    X. Wanget al., “Deep mixture of experts via shallow embedding,” in Uncertainty in artificial intelligence. PMLR, 2020, pp. 552–562

    2020

  34. [42]

    Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks,

    M. N. R. Chowdhuryet al., “Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks,” in International Conference on Machine Learning. PMLR, 2023, pp. 6074–6114

    2023

  35. [43]

    Spatial mixture-of-experts,

    N. Dryden and T. Hoefler, “Spatial mixture-of-experts,”Advances in Neural Information Processing Systems, vol. 35, pp. 11 697–11 713, 2022

    2022

  36. [44]

    Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults,

    D. S. Marcuset al., “Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults,”Journal of Cognitive Neuroscience, vol. 19, no. 9, pp. 1498–1507, 2007

    2007

  37. [45]

    Miccai multi-atlas labeling beyond the cranial vault– workshop and challenge,

    B. Landmanet al., “Miccai multi-atlas labeling beyond the cranial vault– workshop and challenge,” inProc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, vol. 5, 2015, p. 12

    2015

  38. [46]

    Automatic differentiation in pytorch,

    A. Paszkeet al., “Automatic differentiation in pytorch,” 2017

    2017

  39. [47]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Arxiv Preprint Arxiv:1412.6980, 2014

    2014 arXiv

  40. [48]

    Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool,

    A. A. Taha and A. Hanbury, “Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool,”BMC Medical Imaging, vol. 15, no. 1, pp. 1–28, 2015

    2015

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.