Can I read More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing on EtoBox?

More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing by Shaier, Sagi; Pereira, Francisco; von der Wense, Katharina; Hunter, Lawrence E; Jones, Matt is a scholarly article available to read on EtoBox.

What is More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing about?

The evolution of biological neural systems has led to both modularity and sparse coding, which enables energy efficiency and robustness across the diversity of tasks in the lifespan. In contrast, standard neural networks rely on dense, non-specialized architectures, where all model parameters are simultaneously updated to learn multiple tasks, leading to interference. Current sparse neural network approaches aim to alleviate this issue but are hindered by limitations such as 1) trainable gating functions that cause representation collapse, 2) disjoint experts that result in redundant computation and slow learning, and 3) reliance on explicit input or task IDs that limit flexibility and scalability. In this paper we propose Conditionally Overlapping Mixture of ExperTs (COMET), a general deep learning method that addresses these challenges by inducing a modular, sparse architecture with an exponential number of overlapping experts. COMET replaces the trainable gating function used in Sparse Mixture of Experts with a fixed, biologically inspired random projection applied to individual input representations. This design causes the degree of expert overlap to depend on input similarity,

Author

Shaier, Sagi; Pereira, Francisco; von der Wense, Katharina; Hunter, Lawrence E; Jones, Matt

Published

2024

Language

EN