Pith. sign in

REVIEW 4 major objections 6 minor 46 references

Neural Inhibition Improves Dynamic Routing and Mixture of Experts

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adding a globally connected neural inhibition mask to the input of a Mixture-of-Experts router improves accuracy on generic tasks over datasets that mix several unlabeled data types, and reports gains in both vision…

desk verdict A plausible and partially novel gating mechanism, but the experiments don't separate capacity from inhibition and the LM numbers look wrong. read the letter →

arxiv 2507.03221 v1 pith:42BX7RJC submitted 2025-07-03 cs.LG cs.AI

Will Y. Zou , Jennifer Y. Zhang This is my paper
classification cs.LGcs.AI
keywords neuralinhibitionmixtureofexpertsdynamicroutingglobalexpertspecializationroutergatinglanguagemodelingmulti-statisticsdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that dynamic routing models such as Mixture-of-Experts (MoE) are missing a computational primitive that biological brains have: inhibition. It proposes adding an adaptive, globally connected inhibition mask to the input of the MoE router, so that features shared across multiple unlabeled data modes are suppressed and each sample is routed to a more specialized expert path. The claim is that this improves performance on generic tasks over datasets that mix several input types, and the paper verifies it on a vision task (hand-written digits mixed with square patterns) and on language-model word prediction. A sympathetic reader should take away that router input representations, not just expert architectures, are a promising place to intervene.

What carries the argument

The central object is the Global Inhibition Model, an 'all-to-local' connection scheme that computes a learned soft mask over the MoE router input. Its form is $z^\star = z \odot \sigma(G(x) + \sum_i \mathrm{Pr}_i(x_i) + \sum_j \mathrm{maxpool}(\mathrm{Po}_j(x_j^{(k-1)})))$, where $G$ is a gated linear unit on the current layer, $\mathrm{Pr}_i$ are pre-text networks reading earlier-layer activations, and $\mathrm{Po}_j$ are post-text networks reading later-layer activations from the previous optimization step. The post-text term uses max-pooling over batches so the inhibition signal can be broadcast to all samples despite varying batch sizes; this one-step delay avoids recurrent loops. The mechanism's job is to suppress broadly shared, non-discriminative signals in the neuron population so the router can select a specialized expert path for each data mode.

What would settle it

Run the mixed vision task with mini-batches that deliberately sample one data type per batch, so consecutive batches alternate between digits and squares; if global inhibition's accuracy gain over the plain MoE disappears or reverses, the post-text signal is just a batch-correlation artifact. Conversely, feed the post-text network activations from a randomly shuffled, unrelated set of samples instead of the previous batch; if the gain persists, the inhibition signal is not carrying task-relevant information.

Watch

Extended reading notes

Core claim

The central claim is that globally connected neural inhibition improves MoE performance. The paper introduces a Global Inhibition Model that multiplies the router input by a soft sigmoid mask built from three sources: a single-layer gated linear unit, pre-text connections from earlier layers, and post-text connections from later layers whose activations from the previous training iteration are max-pooled over the batch and broadcast. Applied to a vision MoE on a mixed dataset of hand-written digits and number-of-squares patterns, global inhibition reaches 96.7% test accuracy versus 92.3% for the plain MoE and 81.4% for the baseline; on WMT English word prediction it lowers normalized log-likelihood below both the baseline and the MoE. The paper also reports that neurons whose activations are weakly correlated with the input type, the "common" features, receive stronger inhibition, which it interprets as the mechanism that lets the router specialize.

Load-bearing premise

The load-bearing premise is that the post-text inhibition signal, computed from the previous training iteration's activations and max-pooled over the batch, still carries information that is useful for the current batch; if the data stream is non-stationary or the one-step delay makes the signal stale, the reported gains could come from the batching schedule rather than from inhibition itself.

Editorial extensions

If this is right

  • Adding global inhibition to the router input improves test accuracy on the mixed vision task from 92.3% (plain MoE) to 96.7%, with lower variance across random initializations.
  • On WMT English word prediction, global inhibition lowers normalized log-likelihood relative to both the baseline and the MoE on 300k and 1m sentence subsets.
  • Inhibition masks preferentially suppress "common" features shared across data types, which is the proposed mechanism behind better expert specialization.
  • More diverse or global inhibition connections, from random dropout through single-layer gating to pre-text and post-text connections, yield monotonic improvements, suggesting the effect grows with connectivity diversity.
  • In the tested language-model setting, global inhibition improves performance without expert-balancing regularization or auxiliary load-balancing losses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is really about suppressing shared statistics, then global inhibition could serve as a learned alternative to explicit load-balancing losses in large MoE language models, though the paper only tests small models.
  • The one-step-delayed post-text signal makes the method sensitive to the batching schedule; an online moving-average inhibition signal would test whether the gains persist when consecutive batches are less correlated.
  • A direct extension would be to multimodal MoE models, where each modality is an unlabeled "data statistic" and inhibition might replace modality-specific routing heuristics.
  • The Pearson-correlation analysis suggests a crisp test: artificially perturbing the router input by suppressing its most common-feature neurons should reproduce most of the gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes that biologically inspired neural inhibition, implemented as multiplicative sigmoid gating on the router input of a Mixture-of-Experts model, improves dynamic routing. It introduces three inhibition mechanisms: a single gated linear unit, pre-text connections from earlier layers, and post-text connections using a one-step-delayed, max-pooled signal from later layers, plus a combined Global Inhibition Model. The claims are tested on a synthetic mixed MNIST/squares classification task and on two WMT English language-modeling subsets. The vision experiment reports accuracy with standard deviations over five seeds, while the language-model experiment reports normalized log-likelihood values. Section 5 presents an analysis of inhibition activations versus Pearson correlations to input type. The paper concludes that global inhibition improves MoE performance and calls for more research on inhibition-based routing.

Significance. If the central claim is established, the work would contribute a simple, modular addition to MoE routers that could improve specialization without changing the expert architecture. The synthetic two-statistics dataset is a useful testbed, and the delayed post-text connection is an original workaround to avoid recurrent loops while using later-layer information. The vision experiment has a clean setup, five-seed error bars, and a hypothesis-relevant benchmark. However, the current experimental support is substantially weaker than the conclusions: the vision comparisons are confounded with parameter count, the language-model numbers are not physically plausible as reported, and the mechanism analysis is descriptive rather than causal. The significance of the contribution therefore depends on whether these issues can be resolved in revision.

major comments (4)
  1. [Section 4.1, Table 1] The comparisons in Table 1 do not hold the router-input modeling budget fixed, so the reported accuracy gains cannot be attributed to inhibition as a computational primitive. The MoE baseline has no gate, dropout adds no parameters, One-layer I. (GLU) adds one linear-sigmoid layer, and Pre-text and Global add the G, P_ri, and P_oj networks. The gap from MoE (92.3%) to dropout (94.9%) and from dropout (95.9%) to Pre-text (96.6%) is therefore confounded with additional capacity and with the choice of multiplicative gating. A parameter-matched control, such as a gate with the same parameter count but no cross-layer or delayed-batch structure, is needed before the abstract's 'only through inhibition' claim can be supported.
  2. [Section 4.2, Table 2] The normalized log-likelihood values in Table 2 are physically implausible for next-token prediction. Values such as 1.17e-8 and 2.52e-10 over vocabularies of 8,500 and 15,000 tokens would imply that the model assigns near-certain probability to the correct token on almost every test example, which is not achievable with a softmax over that vocabulary under ordinary training and would correspond to degenerate logits. The table also reports no variance or number of seeds. Please specify the exact definition of 'normalized log-likelihood' (e.g., normalized by tokens or characters), report standard errors, and correct the values if they are errors. As written, this table does not provide credible evidence for the language-modeling claim.
  3. [Section 5, Figures 2 and 3] The analysis in Section 5 is descriptive of the fitted model rather than an independent validation of the hypothesis. The figures plot properties of the already-trained inhibition network's own masks (average inhibition activation per neuron) against the Pearson correlation between that neuron's activations and the input-type meta-label. Finding that the learned mask suppresses low-correlation neurons is a restatement of what the model was trained to do, not evidence that inhibition causes better routing. A causal test would compare router specialization or test accuracy when the inhibition mask is enabled versus disabled, or analyze the uninhibited baseline's features under the same metric.
  4. [Section 3.1 and Table 1] The post-text connection is not shown to contribute to the claimed gains. In Table 1, Pre-text Inhibition (96.6% ± 0.23%) and Global Inhibition (96.7% ± 0.34%) are statistically indistinguishable given the reported standard deviations, so the delayed max-pooled post-text signal from Eq. (3) does not demonstrably drive the improvement. To support the 'global' claim, the paper should report a comparison that isolates the post-text component, such as a variant with only post-text inhibition, and should report significance tests or confidence intervals for the differences among the inhibition variants.
minor comments (6)
  1. [Equation (4)] Equation (4) has mismatched parentheses and appears to place the sum of post-text terms outside the sigmoid: z⋆(k) = z(k) ⊙ σ(G(x(k)) + Σ_i P_ri(x_i)) + Σ_j maxpool(P_oj(x_j^(k-1)))). If the intended formula is z⋆ = z ⊙ σ(G(x) + Σ_i P_ri(x_i) + Σ_j maxpool(P_oj(x_j^(k-1)))), please correct the equation and the surrounding prose.
  2. [Section 4.1, Training paragraph] The sentence 'For evaluation, we use a test batch of 5,120' is ambiguous; it should state whether 5,120 is the evaluation batch size or the test set size, and the test set size should be reported explicitly.
  3. [Section 4.2, Model Architecture] The language-model section does not specify the sizes of the G, P_ri, and P_oj networks for the inhibition variants, nor whether the MoE baseline in Table 2 is the plain two-layer transformer or the MoE variant without inhibition. Please clarify these details for reproducibility.
  4. [Section 6.3] Section 6.3 contains a duplicated sentence about routing relying on representation quality and an incomplete sentence 'The representation improves the modeling capability.' Please revise this paragraph.
  5. [References] Zhou et al. (2022a) and Zhou et al. (2022b) reference the same paper; please merge or differentiate them appropriately.
  6. [Abstract] The claim 'Only through inhibition is the routing mechanism able to effectively select neural pathways' is stronger than the evidence presented; please soften it to a hypothesis or qualified conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: performance claims rely on held-out test accuracy; Section 5 is post-hoc interpretation, not a derivation-level circular step.

full rationale

The central empirical claim ('global inhibition improves MoE model performance') is tested on held-out test splits (Table 1 mixed-numbers accuracy, Table 2 WMT normalized log-likelihood), so it is not constructed from the training fit. Equations (1)-(4) define the inhibition masks as architectural operations; they are not derived from, nor equivalent to, the reported accuracies. No load-bearing self-citation appears: the cited works are standard external references (Shazeer et al., Fedus et al., Dauphin et al.), and no uniqueness theorem or prior result by the present authors is invoked to force the model choice. Section 5's analysis plots the trained model's own inhibition activations against feature correlations with the data-type meta-label and interprets the observed relationship as supporting the inhibition hypothesis; this is post-hoc and descriptive rather than a controlled test, but it is not an equation-level reduction of a prediction to its inputs. The Table 1 comparisons do not control parameter count across algorithms, which is a genuine experimental confound for attributing gains to inhibition, but confounds are correctness risks, not circularity. Accordingly no circular step meeting the quoted-reduction standard is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on a few hand-chosen architectural elements and domain assumptions. The most significant are the delayed post-text signal and the max-pooling broadcast, both of which are ad hoc to this paper. The learned inhibition parameters themselves are standard trainable weights, not free constants.

free parameters (3)
  • dropout ratio for random inhibition baseline = 0.75 (best of 0.25, 0.5, 0.75)
    Dropout baseline performance depends on the ratio; the best ratio was selected by tuning, which may favor the later comparisons.
  • post-text inhibition delay = 1 iteration
    The post-text model uses activations from the previous optimization step; the one-step delay is a hand-chosen design that could affect stability.
  • max-pooling broadcast over batch
    To handle varying batch sizes, activations are max-pooled across the previous batch and broadcast; this hand-chosen operation discards per-sample information.
assumptions (4)
  • domain assumption Input data contains multiple distinct unlabeled types of statistics that can be separated by a router.
    Section 3 condition 1; the mixed-numbers dataset is explicitly constructed to satisfy this.
  • ad hoc to paper Previous-iteration activations, max-pooled and broadcast, provide useful inhibitory information without causing recurrent loops.
    Equation (3); the stability and informativeness of this delayed signal are not proven.
  • ad hoc to paper Multiplicative sigmoid gating is an appropriate computational analog of biological inhibition.
    Equations (1)-(4); this is the core modeling choice.
  • standard math Pearson correlation between router neuron activations and (post hoc) data-type labels is a valid measure of feature discriminability for routing.
    Section 5, Equation (5); standard but applied to a meta-label the model was never trained to use.
invented entities (2)
  • Pre-text inhibition network P_ri
    purpose: Generate inhibitory signals for the router input from earlier-layer activations.
    An internal architectural component; no external falsifiable handle beyond the reported benchmarks.
  • Post-text inhibition network P_oj
    purpose: Generate inhibitory signals from later-layer (even loss) activations of the previous training step.
    An internal delayed-feedback component; no external evidence outside the paper's experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Inhibition Improves Dynamic Routing and Mixture of Experts." pith.science (2026). https://pith.science/paper/42BX7RJC

@misc{pith2026250703221,
  author       = {Pith},
  title        = {Pith review of: Neural Inhibition Improves Dynamic Routing and Mixture of Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/42BX7RJC}},
  note         = {Machine review of arXiv:2507.03221}
}
read the original abstract

To be effective, efficient, and diverse, deep learning models need to dynamically choose its architecture based on signals from a population of neurons. We hypothesize dynamic routing models can be improved with neural inhibition in those neural populations. This means signals commonly shared among the various modes of data statistics can be inhibited so that the routing model can choose a specialized expert path for each data sample. Only through inhibition is the routing mechanism able to effectively select neural pathways. We believe this is an under-studied and under-verified implementation methodology for Mixture-of-Experts, dynamic routing, and transformer language models. We provide experimental evidence that the neural inhibition algorithm significantly boosts the performance of general tasks and motivates more effort to be invested in this research direction.

Figures

Figures reproduced from arXiv: 2507.03221 by the authors.

Figure 1
Figure 1. Examples from the mixed-numbers dataset with hand-written digits and number of squares. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. Number of common vs discrimina￾tive features with inhibition thresholds. stronger inhibition (i.e., lower activation values). This supports the hypothesis that the inhibition mechanism prioritizes the suppression of non-discriminative, broadly shared signals, thereby refining the input representation presented to the router [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 24 canonical work pages

  1. [1]

    Adaptive dropout for training deep neural networks

    Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/7b5b23f4aadf9513306bcd59afb6e4c9-Paper.pdf

    work page 2013

  2. [2]

    Efficient Sparse Coding with the Adaptive Locally Competitive Algorithm for Speech Classification

    Soufiyan Bahadi, Eric Plourde, and Jean Rouat. Efficient sparse coding with the adaptive locally competitive algorithm for speech classification, 2024. URL https://arxiv.org/abs/2409.08188

    work page Pith review arXiv 2024

  3. [3]

    Mixtures of experts for audio-visual learning

    Ying Cheng, Yang Li, Junjie He, and Rui Feng. Mixtures of experts for audio-visual learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 219--243. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/fi...

    work page 2024

  4. [4]

    On the representation collapse of sparse mixture of experts, 2022

    Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts, 2022. URL https://arxiv.org/abs/2204.09179

    arXiv 2022

  5. [5]

    An increase of inhibition drives the developmental decorrelation of neural activity

    Marta Chini, Thomas Pfeffer, and Ileana Hanganu-Opatz. An increase of inhibition drives the developmental decorrelation of neural activity. eLife, 11: 0 e78811, 2022. doi:10.7554/eLife.78811. URL https://doi.org/10.7554/eLife.78811

    work page doi:10.7554/elife.78811 2022

  6. [6]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    arXiv 2024

  7. [7]

    Dauphin, Angela Fan, Michael Auli, and David Grangier

    Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, 2017. URL https://arxiv.org/abs/1612.08083

    arXiv 2017

  8. [8]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International conference on machine learning, pages 5547--5569. PMLR, 2022

    2022

Show all 46 references
  1. [9]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL https://arxiv.org/abs/2101.03961

    2022 arXiv

  2. [10]

    Inhibition as a driver of cortical plasticity

    Robert C Froemke. Inhibition as a driver of cortical plasticity. Neuron, 87 0 (3): 0 681--691, 2015

    2015

  3. [11]

    Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. Dropblock: A regularization method for convolutional networks, 2018. URL https://arxiv.org/abs/1810.12890

    2018 arXiv

  4. [12]

    Principles governing the operation of synaptic inhibition in dendrites

    Albert Gidon and Idan Segev. Principles governing the operation of synaptic inhibition in dendrites. Neuron, 75 0 (2): 0 330--341, 2012

    2012

  5. [13]

    Circuit mechanisms of sensory gain control

    Thomas A Hage, Marco A Svirsky, Adrienne L Fairhall, and Jennifer M Groh. Circuit mechanisms of sensory gain control. Trends in Neurosciences, 45 0 (3): 0 202--214, 2022

    2022

  6. [14]

    Fusemoe: Mixture-of-experts transformers for fleximodal fusion

    Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume...

    2024

  7. [15]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  8. [16]

    Edward Suh

    Weizhe Hua, Yuan Zhou, Christopher De Sa, Zhiru Zhang, and G. Edward Suh. Channel gating neural networks, 2019. URL https://arxiv.org/abs/1805.12549

    2019 arXiv

  9. [17]

    Isaacson and Massimo Scanziani

    Jeffry S. Isaacson and Massimo Scanziani. How inhibition shapes cortical activity. Neuron, 72 0 (2): 0 231--243, 2011. ISSN 0896-6273. doi:https://doi.org/10.1016/j.neuron.2011.09.027. URL https://www.sciencedirect.com/science/article/pii/S0896627311008798

    2011 doi

  10. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  11. [19]

    Kingma, Tim Salimans, and Max Welling

    Diederik P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick, 2015. URL https://arxiv.org/abs/1506.02557

    2015 arXiv

  12. [20]

    Neuronal diversity and temporal dynamics: the unity of hippocampal circuit operations

    Thomas Klausberger and Peter Somogyi. Neuronal diversity and temporal dynamics: the unity of hippocampal circuit operations. Science, 321 0 (5885): 0 53--57, 2008

    2008

  13. [21]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. CoRR, abs/2006.16668, 2020. URL https://arxiv.org/abs/2006.16668

    2006 arXiv

  14. [22]

    Dropkey, 2023

    Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey, 2023. URL https://arxiv.org/abs/2208.02646

    2023 arXiv

  15. [23]

    Uni-moe: Scaling unified multimodal llms with mixture of experts

    Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. Uni-moe: Scaling unified multimodal llms with mixture of experts. arXiv preprint arXiv:2405.11273, 2024

    2024 arXiv

  16. [24]

    Patchdropout: Economizing vision transformers using patch dropout, 2022

    Yue Liu, Christos Matsoukas, Fredrik Strand, Hossein Azizpour, and Kevin Smith. Patchdropout: Economizing vision transformers using patch dropout, 2022. URL https://arxiv.org/abs/2208.07220

    2022 arXiv

  17. [25]

    Findings of the WMT 2024 shared task of the open language data initiative

    Jean Maillard, Laurie Burchell, Antonios Anastasopoulos, Christian Federmann, Philipp Koehn, and Skyler Wang. Findings of the WMT 2024 shared task of the open language data initiative. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ni...

    2024 doi

  18. [26]

    Interneurons of the neocortical inhibitory system

    Henry Markram, Maria Toledo-Rodriguez, Yun Wang, Anirudh Gupta, Gilad Silberberg, and Caizhi Wu. Interneurons of the neocortical inhibitory system. Nature reviews neuroscience, 5 0 (10): 0 793--807, 2004

    2004

  19. [27]

    Evaluation of adaptive mixtures of competing experts

    Steven Nowlan and Geoffrey E Hinton. Evaluation of adaptive mixtures of competing experts. In R.P. Lippmann, J. Moody, and D. Touretzky, editors, Advances in Neural Information Processing Systems, volume 3. Morgan-Kaufmann, 1990. URL https://proceedings.neurips.cc/paper_files/...

    1990

  20. [28]

    Rozell, D

    Christopher J. Rozell, D. H. Johnson, Richard G. Baraniuk, and Bruno A. Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural Computation, 20 0 (10): 0 2526--2563, 2008. doi:10.1162/neco.2008.03-07-486

    2008 doi

  21. [29]

    Gabaergic inhibition in visual cortical plasticity

    Alessandro Sale, Nicoletta Berardi, Maria Spolidoro, Laura Baroncelli, and Lamberto Maffei. Gabaergic inhibition in visual cortical plasticity. Frontiers in cellular neuroscience, 4: 0 1534, 2010

    2010

  22. [30]

    Spencer Shapero, Ming Zhu, Paul Hasler, and Christopher J. Rozell. Optimal sparse approximation with integrate and fire neurons. International Journal of Neural Systems, 24 0 (5): 0 1440001, Aug 2014. doi:10.1142/S0129065714400012. Epub 2014 Mar 23

    2014 doi

  23. [31]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020. URL https://arxiv.org/abs/2002.05202

    2020 arXiv

  24. [32]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701.06538

    2017 arXiv

  25. [33]

    Mome: Mixture of multimodal experts for generalist multimodal large language models, 2024

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models, 2024. URL https://arxiv.org/abs/2407.12709

    2024 arXiv

  26. [34]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 0 (56): 0 1929--1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html

    1929

  27. [35]

    Highway networks

    Rupesh Kumar Srivastava, Klaus Greff, and J \" u rgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015. URL http://arxiv.org/abs/1505.00387

    2015 arXiv

  28. [36]

    Staudemeyer and Eric Rothstein Morris

    Ralf C. Staudemeyer and Eric Rothstein Morris. Understanding LSTM - a tutorial into long short-term memory recurrent neural networks. CoRR, abs/1909.09586, 2019. URL http://arxiv.org/abs/1909.09586

    1909 arXiv

  29. [37]

    Emergence of a'visual number sense'in hierarchical generative models

    Ivilin Stoianov and Marco Zorzi. Emergence of a'visual number sense'in hierarchical generative models. Nature neuroscience, 15 0 (2): 0 194--196, 2012

    2012

  30. [38]

    Gabaergic interneurons in the neocortex: from cellular properties to circuits

    R \'e mi Tremblay, Sophia Lee, and Bernardo Rudy. Gabaergic interneurons in the neocortex: from cellular properties to circuits. Neuron, 91 0 (2): 0 260--292, 2016

    2016

  31. [39]

    Interneuron circuits in the mouse neocortex are differentially engaged by whisker touch and locomotion

    Maria Udakis, Victor Pedrosa, Simon Chamberland, Laurent Tyan, Corrado Cal , and Carl CH Petersen. Interneuron circuits in the mouse neocortex are differentially engaged by whisker touch and locomotion. Cell Reports, 30 0 (11): 0 3274--3289, 2020

    2020

  32. [40]

    Diverse and specific patterns of gabaergic interneuron connectivity in the neocortex

    Joanna Urban-Ciecko and Alison L Barth. Diverse and specific patterns of gabaergic interneuron connectivity in the neocortex. Current Opinion in Neurobiology, 63: 0 86--92, 2020

    2020

  33. [41]

    Regularization of neural networks using dropconnect

    Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058--1066. PMLR, 2013

    2013

  34. [42]

    Division and subtraction by distinct cortical inhibitory networks in vivo

    Neil R Wilson, Caroline A Runyan, Flora L Wang, and Mriganka Sur. Division and subtraction by distinct cortical inhibitory networks in vivo. Nature, 488 0 (7411): 0 343--348, 2012

    2012

  35. [43]

    Sebastian Seung

    Xiaohui Xie, Richard Hahnloser, and H. Sebastian Seung. Learning winner-take-all competition between groups of neurons in lateral inhibitory networks. In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13. MIT Press, 200...

    2000

  36. [44]

    Revisiting structured dropout, 2022

    Yiren Zhao, Oluwatomisin Dada, Xitong Gao, and Robert D Mullins. Revisiting structured dropout, 2022. URL https://arxiv.org/abs/2210.02570

    2022 arXiv

  37. [45]

    Mixture-of-experts with expert choice routing, 2022 a

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing, 2022 a . URL https://arxiv.org/abs/2202.09368

    2022 arXiv

  38. [46]

    Mixture-of-experts with expert choice routing

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35: 0 7103--7114, 2022 b

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.