Can I read WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting on EtoBox?

WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting by Liang, Aobo; Sun, Yan; Guizani, Nadra is a scholarly article available to read on EtoBox.

What is WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting about?

In recent years, Transformer-based models (Transformers) have achieved significant success in multivariate time series forecasting (MTSF). However, previous works focus on extracting features either from the time domain or the frequency domain, which inadequately captures the trends and periodic characteristics. To address this issue, we propose a wavelet learning framework to model complex temporal dependencies of the time series data. The wavelet domain integrates both time and frequency information, allowing for the analysis of local characteristics of signals at different scales. Additionally, the Softmax self-attention mechanism used by Transformers has quadratic complexity, which leads to excessive computational costs when capturing long-term dependencies. Therefore, we propose a novel attention mechanism: Rotary Route Attention (RoRA). Unlike Softmax attention, RoRA utilizes rotary position embeddings to inject relative positional information to sequence tokens and introduces a small number of routing tokens $r$ to aggregate information from the $KV$ matrices and redistribute it to the $Q$ matrix, offering linear complexity. We further propose WaveRoRA, which leverages RoRA

Author

Liang, Aobo; Sun, Yan; Guizani, Nadra

Published

2024

Language

EN