Monet composes experts from shared sub-layers to scale to 262,144 experts per layer with parameter growth proportional to the square root of the expert count, yielding more monosemantic and manipulable LLMs.
F**kyou!F**k (...)* (16.68%)(...)Snakesonamotherf*ckingplane
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.AI 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Monet: Mixture of Monosemantic Experts for Transformers
Monet composes experts from shared sub-layers to scale to 262,144 experts per layer with parameter growth proportional to the square root of the expert count, yielding more monosemantic and manipulable LLMs.