Pith. sign in

REVIEW 5 major objections 5 minor 42 references

Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that Soft MoE dispatch weights already form segmentation-like maps, and that steering them toward semantic foreground with an auxiliary loss plus a zero-initialized LayerScale yields more accurate and interpretable…

desk verdict Plausible and novel idea for Soft MoE routing, but the headline gain is not cleanly attributable to the semantic prior — the LayerScale/residual change carries much of the weight. read the letter →

arxiv 2505.18586 v1 pith:2MNO2QYR submitted 2025-05-24 cs.CV

Chengxi Min , Wei Wang , Yahui Liu , Weixin Ye , Enver Sangineto , Qi Wang , Yao Zhao This is my paper
classification cs.CV
keywords MixtureofExpertsSoftMoEsemanticforegroundpriorauxiliaryroutinglossdispatchweightsLayerScalevisiontransformerexpertinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that Soft MoE—a vision architecture where every token is softly assigned to every expert through continuous dispatch weights—already produces spatial patterns that look like object outlines, and that this implicit structure can be exploited. It adds an auxiliary loss that pushes the dispatch weights to overlap with a foreground mask produced by an external detector–segmenter pipeline, and it inserts a zero-initialized LayerScale into the last residual path so that the extra supervision reaches the deepest routing layer. The reported effect is a top-1 accuracy gain from 73.9 to 74.5 on ImageNet-1K and from 75.4 to 76.8 on ImageNet-100, plus gains of +0.5 to +2.8 accuracy points when fine-tuning on four smaller datasets. A sympathetic reader would care because the changes are cheap, architecture-agnostic, and also make the routing visibly more interpretable: experts concentrate on the main object and specialize on different parts of it.

What carries the argument

The machinery is the dispatch weight matrix $D$ of the last Soft MoE layer, collapsed into a per-token mean $W$, binarized against its own mean to form $B$, and compared with the prior mask $M$ via a soft IoU-style score $p$; the loss $L_{\mathrm{aux}}=-\log(p+\epsilon)$ is what actually steers routing. The second piece is a vector LayerScale $\gamma$ initialized to zero on the final residual connection, $y'=r+\gamma\odot x$, which lets the auxiliary gradient influence the deepest expert layer without disturbing early optimization. The paper also treats the choice of mask type and layer placement as part of the mechanism: foreground masks at the 8th layer work, while background masks or 7th-layer supervision degrade accuracy, which the authors read as evidence that semantic alignment, not any spatial target, is what helps.

What would settle it

Take the same training setup and replace the semantic foreground masks with random binary masks matched in area; if accuracy stays at or above the reported 74.5, then alignment with semantic foreground is not the cause of the gain. A second check is to train on a dataset where the class label does not correspond to the visual foreground, such as scene images with small objects, and see whether the auxiliary loss still helps or starts to hurt.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the averaged dispatch weights of a Soft MoE layer form a spatial saliency map, and that this map can be trained to agree with semantic foreground masks through a differentiable overlap score. With $W_i$ the mean over experts and slots of the dispatch weight $D_{i,j}$ for token $i$, the paper thresholds $W$ at its own average to obtain a binary mask $B$, then compares $B$ with the external foreground mask $M$ through the intersection $O_i=B_i\cap M_i$ and union $U_i=B_i\cup M_i$. The importance alignment score $p=\sum_i W_iO_i/\sum_i W_iU_i$ is turned into the auxiliary loss $L_{\mathrm{aux}}=-\log(p+\epsilon)$, added to the classification loss with weight $\lambda=0.01$. The second proposed change is the final-layer residual $y'=r+\gamma\odot x$, where the LayerScale vector $\gamma$ starts at zero so the residual contribution grows only as training proceeds. In the paper's telling, these two changes make the dispatch weights concentrate on foreground, raise accuracy on every benchmark tested, and make individual experts specialize on different foreground subregions.

Load-bearing premise

The load-bearing premise is that the externally produced foreground masks are accurate enough for every training image that pushing dispatch weights to overlap with them improves routing; if the masks are noisy, or if the mean-thresholded average of dispatch weights is not a faithful saliency map, the auxiliary loss steers experts toward the wrong regions.

Editorial extensions

If this is right

  • If the central claim is correct, any Soft MoE vision model can adopt the two modifications for near-zero extra compute and gain accuracy; the paper reports +0.6 top-1 on ImageNet-1K and +1.4 on ImageNet-100 after 100 epochs.
  • The loss also works when applied only during fine-tuning, so the benefit is not limited to pretraining; reported fine-tuning gains range from +0.5 to +2.8 accuracy points across Stanford Cars, Clipart, Painting, and Sketch.
  • Expert routing becomes more interpretable: average dispatch maps align with the foreground object, and individual experts specialize on distinct foreground parts instead of clustering on background.
  • The recommended configuration is foreground supervision at the last expert layer; the paper reports that background masks, earlier-layer placement, and simultaneous 7th- and 8th-layer supervision all hurt, which it attributes to conflicting signals and loss shortcutting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If dispatch-weight maps stay foreground-aligned after training, they could be reused as a free saliency or weakly supervised segmentation signal at inference, removing the need for external masks.
  • The mask-dependence admits a direct stress test: replacing the external detector–segmenter masks with coarse boxes, randomly corrupted masks, or self-supervised saliency estimates would show how much of the gain is true semantic alignment versus any spatially consistent regularizer.
  • The same foreground-aligned auxiliary loss should transfer to hard Top-K MoE routers or dense attention layers, because the mechanism only requires a spatial routing or attention map and a residual connection, not soft dispatch specifically.
  • The authors' 'loss shortcutting' observation implies that multi-layer auxiliary supervision needs explicit coordination; a schedule or weighting that prevents the 7th layer from absorbing all supervision could extend the gains beyond single-layer guidance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a method to improve Soft MoE routing in vision transformers by introducing a foreground-guided auxiliary loss and a LayerScale residual connection. The auxiliary loss (Eq. 9) encourages the average dispatch weights (Eq. 4) to align with foreground masks extracted offline by Grounding DINO and SAM, using a thresholded binary mask (Eq. 5) and a soft IoU-like overlap score (Eq. 8). The LayerScale mechanism (Eq. 11) modulates the skip connection in the last block. Experiments on ImageNet-1K, ImageNet-100, and four fine-tuning datasets report improvements of +0.6, +1.4, and +0.5 to +2.8 top-1 accuracy, respectively, along with qualitative evidence of more interpretable routing. The paper also includes ablations on loss weight, mask type, and LayerScale variants.

Significance. If the reported gains are reproducible, the method is a simple, low-cost addition to Soft MoE with no extra inference overhead, and it provides qualitative evidence that routing can be made more semantically interpretable. The code and mask-generation pipeline are promised publicly. However, the empirical evidence is currently too thin to support the central attribution claim: all results are single runs, the margins on ImageNet-1K are small, and the ablation (Tables 3-4) shows that the auxiliary loss alone slightly degrades accuracy while the LayerScale/residual modification alone accounts for most of the gain. The novelty is moderate, as the approach combines known components (segmentation-based supervision and LayerScale) in a new context. With stronger statistical evidence and a cleaner isolation of the semantic-prior effect, the paper could be a useful contribution to MoE routing.

major comments (5)
  1. [§5.2, Tables 3-4] The ablation does not support the claim that the semantic foreground prior is the main source of the improvement. Aux loss alone yields 73.8 (worse than the 73.9 baseline), LayerScale alone yields 74.0, removing the skip connection yields 74.3, and the full method yields 74.5. This means the +0.6 over baseline can be largely explained by the residual/LayerScale modification, with the semantic loss contributing only in combination. To isolate the semantic prior, please add an ablation that holds the LayerScale fixed and compares (a) no auxiliary loss, (b) the auxiliary loss with random masks, and (c) the auxiliary loss with background masks, each run with multiple seeds.
  2. [§5.1 and Appendix B] Hyperparameters such as the auxiliary loss weight λ (Eq. 10), the mask type (foreground vs. background), and the placement at the 8th layer were selected using the same ImageNet-1K benchmark that is later used for the headline result (Table 1). This creates a risk of selection bias. Please either use a held-out validation split for model selection or fix the protocol a priori and report results on a different large-scale benchmark to confirm that the gains are not a result of tuning on the test distribution.
  3. [§4, Eq. (5)] Because each column of the dispatch matrix D is normalized over tokens (Eq. 2), the mean of the average dispatch weights W is always w = 1/m. The binarization in Eq. (5) therefore encodes only 'above-average' tokens and can systematically miss small or multi-object foreground regions whose total weight is diluted by background. This may explain the weak performance of the auxiliary loss alone. The paper should justify this threshold, test alternatives (e.g., top-k tokens, a fixed percentile, or a continuous weighting), and report how often the resulting mask B meaningfully overlaps the foreground.
  4. [Tables 1-5] All accuracy results are reported from single runs without standard deviations, confidence intervals, or significance tests. Given that the ImageNet-1K improvements are at the level of 0.1-0.6 percentage points, these differences may fall within run-to-run noise. Please provide results averaged over at least three independent seeds (mean ± std) and, if feasible, a paired significance test for the main comparisons.
  5. [§6, Limitation and Algorithm 1] The manuscript acknowledges that performance depends on mask quality but does not quantify this dependence. The auxiliary loss requires reliable foreground masks for every training image, yet the paper reports no statistics on mask coverage, no fraction of samples filtered out as invalid (Algorithm 1), and no sensitivity analysis to mask perturbations. Please add an analysis of mask quality and a robustness experiment (e.g., using degraded or random masks) to demonstrate that the semantic prior is actually the active ingredient.
minor comments (5)
  1. [Abstract and §1] There are typos: 'phrases' should be 'phases' in §1, and 'avialable' should be 'available' in the Abstract.
  2. [§4] The text contains typos: 'Grouding DINO' should be 'Grounding DINO', and 'shwon' should be 'shown'. In the same section, 'Multi-Scale Attention' likely should be 'Multi-Head Self-Attention' when referring to the MSA layer.
  3. [§5.2] The text 'When training from scartch' should be 'When training from scratch'.
  4. [Appendix D] The sentence 'We also explore combining different losses across layers in figure 8' appears to refer to Figure 7, which shows the combination settings of Table 5; Figure 8 shows LayerScale variants. Also, the caption of Figure 7(d) refers to '(3)' but should refer to '(c)'.
  5. [§4, Eq. (11)] The rendered notation 'γγγ' is confusing; it should be a single vector-valued parameter γ (or a bold γ) consistently with the text 'learnable parameters in the LayerScale module'.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-level circularity: the auxiliary loss is a training objective, not a re-statement of the reported accuracy, and the evaluation is external.

full rationale

The paper's chain is empirical rather than derivational. Eq. (9) defines an auxiliary loss from a soft overlap (Eq. 8) between mean-thresholded dispatch weights (Eq. 5) and externally produced foreground masks; the headline evidence is held-out top-1 accuracy in Tables 1-2, which is not a term in that loss, so the main result does not reduce to the loss by construction. The baseline is independently reproduced from the Soft MoE codebase [10]/[29], and the comparison is against an external benchmark suite. The only same-author citation ([28]) is used as inspiration for exploiting skip connections; the actual LayerScale mechanism is attributed to external work [35] and is validated by the paper's own ablations (Tables 3-4), so the citation is not load-bearing. The ablation also reports that the auxiliary loss alone slightly hurts (73.8 vs 73.9), which is inconsistent with the headline gain being a forced artifact of the loss definition. Two non-circularity caveats remain: hyperparameters (lambda, mask type, placement) were selected on ImageNet-1K validation and then the same dataset's test accuracy is reported, which is a mild selection-on-benchmark effect; and the visual 'interpretability' claim is partly tautological because the loss directly optimizes the overlap that is later visualized. Neither is an equation-level equivalence between predicted output and input, so no circular step is established.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. It depends on two categories of assumptions: the quality of external segmentation masks and the validity of mean-threshold dispatch weights as routing saliency. The main free parameter is the auxiliary loss weight lambda=0.01, tuned on ImageNet-1K validation; the binarization threshold is also a hand-chosen design. The LayerScale init is a design choice inherited from CaiT.

free parameters (2)
  • Auxiliary loss weight lambda = 0.01
    Selected as best on ImageNet-1K validation accuracy (Figure 5). This is the main knob of the method and is tuned on the same benchmark used for the headline result.
  • Binarization threshold for dispatch mask = Sample mean of W (Eq. 5)
    Hand-chosen global threshold per sample; no learned or cross-validated value. Different thresholds could change the loss landscape and the resulting routing behavior.
assumptions (4)
  • domain assumption Foreground masks produced by Grounding DINO + SAM correctly identify the semantically relevant object regions for the training images.
    The auxiliary loss (Eq. 9) is a direct function of these masks. If masks are wrong, the loss pushes dispatch weights toward incorrect regions. Acknowledged as a limitation in the conclusion.
  • domain assumption Mean dispatch weight per token, averaged over all experts and slots (Eq. 4), is a faithful spatial saliency map of routing behavior.
    The method builds the binary mask B exclusively from this average. Averaging may blur per-expert specialization, yet the loss treats this average as the routing signal to align with the foreground.
  • domain assumption The binarization operation (Eq. 5) is treated as a fixed transformation in the backward pass.
    Gradients do not flow through the threshold; this is a straight-through estimator. The paper does not discuss this approximation or its effect on optimization.
  • domain assumption LayerScale with zero init does not harm convergence and can be safely added to the last residual connection.
    Empirically validated in Table 4, but no theoretical justification. Different initialization or placement could alter results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing." pith.science (2026). https://pith.science/paper/2MNO2QYR

@misc{pith2026250518586,
  author       = {Pith},
  title        = {Pith review of: Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MNO2QYR}},
  note         = {Machine review of arXiv:2505.18586}
}
read the original abstract

Mixture-of-Experts (MoE) models have emerged as a promising direction for scaling vision architectures efficiently. Among them, Soft MoE improves training stability by assigning each token to all experts via continuous dispatch weights. However, current designs overlook the semantic structure which is implicitly encoded in these weights, resulting in suboptimal expert routing. In this paper, we discover that dispatch weights in Soft MoE inherently exhibit segmentation-like patterns but are not explicitly aligned with semantic regions. Motivated by this observation, we propose a foreground-guided enhancement strategy. Specifically, we introduce a spatially aware auxiliary loss that encourages expert activation to align with semantic foreground regions. To further reinforce this supervision, we integrate a lightweight LayerScale mechanism that improves information flow and stabilizes optimization in skip connections. Our method necessitates only minor architectural adjustments and can be seamlessly integrated into prevailing Soft MoE frameworks. Comprehensive experiments on ImageNet-1K and multiple smaller-scale classification benchmarks not only showcase consistent performance enhancements but also reveal more interpretable expert routing mechanisms.

Figures

Figures reproduced from arXiv: 2505.18586 by the authors.

Figure 1
Figure 1. Visualization of dispatch weight maps during training from scratch at different epochs (20, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The process for generating the foreground mask involves leveraging Grounding DINO in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of our proposed method. We first compute the average dispatch weights from the Soft MoE module and apply thresholding based on their mean value to generate a binary weight mask. We encourage this weight mask to overlap with the prior foreground mask as much as possible, guiding expert attention toward semantically meaningful regions in the image. Additionally, we introduce a LayerScale module with an initia… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Visualization of dispatch weight maps under different ablation settings. Colors range from [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Effect of loss weight λ on accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: (a) Validation accuracy curves during training. The model trained with our proposed routing [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Visualization of dispatch weights under different auxiliary loss, corresponding to the [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Visualization of dispatch weights under different LayerScale configurations in the last Soft [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Visualization of individual expert activations from the 8th Soft MoE layer. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Dispatch weight visualizations on multi-object images. (a) and (b) compare the baseline [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 26 canonical work pages

  1. [28]

    Spatial entropy as an inductive bias for vision transformers.Machine Learning, 113(9):6945– 6975, 2024

    Elia Peruzzo, Enver Sangineto, Yahui Liu, Marco De Nadai, Wei Bi, Bruno Lepri, and Nicu Sebe. Spatial entropy as an inductive bias for vision transformers.Machine Learning, 113(9):6945– 6975, 2024. 1, 6

    work page 2024

  2. [1]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InEuropean Conference on Computer Vision (ECCV), 2020. 2

    2020

  3. [2]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 1, 2

    work page 2021

  4. [3]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 1, 2, 6, 13

    arXiv 2010

  5. [4]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022. 2

    2022

  6. [5]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 6

    work page 2016

  7. [6]

    Tutel: Adaptive mixture-of-experts at scale.Proceedings of Machine Learning and Systems, 5:269–287, 2023

    Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at scale.Proceedings of Machine Learning and Systems, 5:269–287, 2023. 3

    work page 2023

  8. [7]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024. 2

    arXiv 2024

Show all 42 references
  1. [8]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020. 1, 2

    2001 arXiv

  2. [9]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2, 4

    2023

  3. [10]

    Soft mixture of experts - official implementation

    Forest Kodom. Soft mixture of experts - official implementation. https://github.com/ fkodom/soft-mixture-of-experts, 2023. Accessed: 2025-05-16. 13

    2023

  4. [11]

    3d object representations for fine- grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. InProceedings of the IEEE International Conference on Computer Vision Workshops, 2013. 6, 12

    2013

  5. [12]

    Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks.Advances in neural information processing systems, 25, 2012. 1

    2012

  6. [13]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020. 2

    2006 arXiv

  7. [14]

    Base layers: Simplifying training of large, sparse models

    Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. InInternational Conference on Machine Learning (ICML), 2021. 2

    2021

  8. [15]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4

    2022

  9. [16]

    Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704, 2020

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training.arXiv preprint arXiv:2006.15704, 2020. 13 10

    2006 arXiv

  10. [17]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024. 1

    2024 arXiv

  11. [18]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. 1

    2024 arXiv

  12. [19]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision (ECCV), 2024. 4

    2024

  13. [20]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2

    2021

  14. [21]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  15. [22]

    Object-centric learning with slot attention.Advances in Neural Information Processing Systems (NeurIPS), 2020

    Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention.Advances in Neural Information Processing Systems (NeurIPS), 2020. 2

    2020

  16. [23]

    Language segment anything

    Luca Medeiros. Language segment anything. https://github.com/luca-medeiros/ lang-segment-anything, 2023. Accessed: 2025-05-16. 4

    2023

  17. [24]

    Soft merging of experts with adaptive routing.arXiv preprint arXiv:2306.03745, 2023

    Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing.arXiv preprint arXiv:2306.03745, 2023. 1, 2

    2023 arXiv

  18. [25]

    Intriguing properties of vision transformers

    Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems (NeurIPS), 2021. 2

    2021

  19. [26]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023. 1

    2023 arXiv

  20. [27]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 6, 12

    2019

  21. [29]

    From sparse to soft mixtures of experts.arXiv preprint arXiv:2308.00951, 2023

    Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts.arXiv preprint arXiv:2308.00951, 2023. 1, 2, 3, 7

    2023 arXiv

  22. [30]

    Scaling vision with sparse mixture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems (NeurIPS), 2021. 1, 2, 3, 13

    2021

  23. [31]

    Hash layers for large sparse models

    Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems (NeurIPS), 2021. 2

    2021

  24. [32]

    Imagenet large scale visual recognition challenge.International Journal of Computer Vision (IJCV), 115:211–252, 2015

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.International Journal of Computer Vision (IJCV), 115:211–252, 2015. 6, 12

    2015

  25. [33]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017. 2, 3 11

    2017 arXiv

  26. [34]

    Contrastive multiview coding

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. InEuropean Conference Computer Vision (ECCV), 2020. 6, 12

    2020

  27. [35]

    Go- ing deeper with image transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2, 6

    2021

  28. [36]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 4

    2017

  29. [37]

    Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self- supervised transformer and normalized cut.IEEE Transactions on Pattern Analysis and Machine Intelligence...

    2023

  30. [38]

    Mixture of lora experts.arXiv preprint arXiv:2404.13628, 2024

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts.arXiv preprint arXiv:2404.13628, 2024. 1, 2

    2024 arXiv

  31. [39]

    Scaling vision transform- ers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1

    2022

  32. [40]

    Lory: Fully differentiable mixture- of-experts for autoregressive language model pre-training.arXiv preprint arXiv:2405.03133,

    Zexuan Zhong, Mengzhou Xia, Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture- of-experts for autoregressive language model pre-training.arXiv preprint arXiv:2405.03133,

    arXiv

  33. [41]

    ibot: Image bert pre-training with online tokenizer.arXiv preprint arXiv:2111.07832, 2021

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer.arXiv preprint arXiv:2111.07832, 2021. 2

    2021 arXiv

  34. [42]

    scale + linear

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems (NeurIPS), 2022. 2 A Implementation of Aux Loss Algorithm 1 outl...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.