| Elad Hazan |  | Thursday March 20thThursday | Spectral Transformers | We'll discuss a new technique for sequence modeling for prediction tasks with long range dependencies and fast generation. At the heart of the method is a new formulation for state space models (SSMs) based on learning linear dynamical systems with the spectral filtering algorithm. This gives rise to a novel sequence prediction architecture we call a spectral state space model. Spectral state space models have two primary advantages. First, they have provable robustness properties as their performance depends on neither the spectrum of the underlying dynamics nor the dimensionality of the problem. Second, these models are constructed with fixed convolutional filters that do not require learning while still outperforming SSMs in both theory and practice. The resulting models are evaluated on synthetic dynamical systems as well as long-range prediction tasks of various modalities. These evaluations support the theoretical benefits of spectral filtering for tasks requiring very long range memory. We will also discuss recent work showing fast generation and provable length generalization of these models. More information about spectral transformers and filtering |
