Multivariate long-term series forecasting method with DFT-based frequency-sensitive dual-branch Transformer

doi:10.11772/j.issn.1001-9081.2023091320

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (9): 2739-2746.DOI: 10.11772/j.issn.1001-9081.2023091320

• Data science and technology • Previous Articles Next Articles

Multivariate long-term series forecasting method with DFT-based frequency-sensitive dual-branch Transformer

Liehong REN¹, Lyuwen HUANG¹(), Xu TIAN¹, Fei DUAN²

^1.College of Information Engineering，Northwest A&F University，Xianyang Shaanxi 712100，China
^2.Ningbo Industrial Internet Institute，Ningbo Zhejiang 315000，China

Received:2023-09-26 Revised:2023-11-14 Accepted:2023-11-20 Online:2023-12-01 Published:2024-09-10
Contact: Lyuwen HUANG
About author:REN Liehong， born in 1997， M. S. candidate. His research interests include time series prediction， deep learning.
TIAN Xu， born in 1998， M. S. candidate. His research interests include embedded system， non-destructive detection of fruit quality.
DUAN Fei， born in 1978， Ph. D. His research interests include MEMS sensor， AI sensing.
Supported by:
National Key Research and Development Program of China(2020YFD1100601)

基于DFT的频率敏感双分支Transformer多变量长时间序列预测方法

任烈弘¹, 黄铝文¹(), 田旭¹, 段飞²

^1.西北农林科技大学信息工程学院，陕西咸阳 712100
^2.宁波工业互联网研究院，浙江宁波 315000

通讯作者: 黄铝文
作者简介:任烈弘（1997—），男，山西吕梁人，硕士研究生，CCF会员，主要研究方向：时间序列预测、深度学习
黄铝文（1976—），男，湖南湘乡人，副教授，博士，CCF会员，主要研究方向：生物图像处理、机器人
田旭（1998—），男，甘肃天水人，硕士研究生，主要研究方向：嵌入式系统、水果品质无损检测
段飞（1978—），男，湖南湘乡人，博士，主要研究方向：MEMS传感器、AI感知。
基金资助:
国家重点研发计划项目(2020YFD1100601)

Abstract

Abstract:

In multivariate long-term time series forecasting， only relying on time domain analysis often falls to capture long time-series dependencies， leading to insufficient information utilization and not high enough prediction accuracy. To solve these problems， combined with time and frequency domain analyses， a Frequency-Sensitive Dual-branch Transformer with Discrete Fourier Transform （DFT） for multivariate long-term series forecasting （FSDformer） method was proposed. Firstly， by utilizing DFT， the transformation between time and frequency was accomplished， allowing the decomposition of complex time-series data into three structurally simple components： low-frequency trend item， medium-frequency seasonal item， and high-frequency residual item. Then， a dual-branch structure was adopted： one branch dedicated to predict medium- and high-frequency components， with an Encoder-Decoder structure applied to design a periodic enhancement attention mechanism， and another dedicated forecast to low-frequency trend components， with a MultiLayer Perceptron （MLP） structure. Finally， the prediction results from both branches were aggregated to obtain the final multivariate long-term time series forecasting results. FSDformer was compared with five classical algorithms on two datasets. On the Electricity dataset， when the historical sequence length is 96 and the predicted sequence length is 336， compared to the comparison algorithms such as Autoformer， FSDformer decreases the Mean Absolute Error （MAE） by 11.5%-29.1%， and decreases the Mean Square Error （MSE） by 20.9%-43.7%， reaching the optimal prediction accuracy. Experimental results show that， FSDformer can capture the dependencies within long-term time series data efficiently， and can improve the prediction stability of model while enhancing prediction accuracy and computational efficiency.

Key words: Discrete Fourier Transform (DFT), frequency-sensitive, time-series prediction, series decomposition, Transformer, periodic enhancement attention

摘要：

在进行多变量长时间序列预测时，仅利用时域分析通常无法充分捕捉长时间序列依赖，而这会导致信息利用率不足、预测精度不够高。因此，结合频域时域分析，提出一种基于离散傅里叶变换（DFT）的频率敏感双分支多变量长时间序列预测（FSDformer）方法。首先，通过DFT实现时间和频率的相互转换，从而将复杂的时间序列数据分解为结构简单的低频趋势项、中频季节项和高频余项3个分量；其次，采用双分支结构，针对中高频分量预测，应用Encoder-Decoder结构，设计了周期性增强注意力机制；针对低频趋势分量预测，采用多层感知机（MLP）结构；最后将中高频分量与低频分量预测结果相加，得到多变量长时间序列的最终预测结果。在2个数据集上把FSDformer与其他5个经典算法进行了对比分析，在Electricity数据集上，当历史序列长度为96，预测序列长度为336时，相较于Autoformer等对比算法，FSDformer的平均绝对误差（MAE）下降了11.5%~29.1%，均方误差（MSE）下降了20.9%~43.7%，达到了最优预测精度。实验结果表明，FSDformer能有效捕捉长时间序列的相关依赖，在提升预测精度和计算效率的同时，增强了模型预测的稳定性。

关键词: 离散傅里叶变换, 频率敏感, 时间序列预测, 序列分解, Transformer, 周期性增强注意力

CLC Number:

TP391

Liehong REN, Lyuwen HUANG, Xu TIAN, Fei DUAN. Multivariate long-term series forecasting method with DFT-based frequency-sensitive dual-branch Transformer[J]. Journal of Computer Applications, 2024, 44(9): 2739-2746.

任烈弘, 黄铝文, 田旭, 段飞. 基于DFT的频率敏感双分支Transformer多变量长时间序列预测方法[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2739-2746.

Figures/Tables 17

Fig. 1 Structure of FSDformer model

Fig. 2 Prediction process of low-frequency trend term

Fig. 3 Structure of time-series decomposition

Fig. 4 Periodic aggregation

Fig. 5 Mechanism of periodic enhancement

Fig. 6 Encoder-Decoder structure with periodic enhancement

Fig. 7 Prediction flow of FSDformer

Tab. 1 Basic information of datasets

数据集	采样时间间隔/h	数据维度	时间步
Electricity	1	321	26 304
Traffic	1	862	17 544

Fig. 8 Curve of electricity consumption over time （partial）

Tab. 2 Prediction results of multivariate time-series with different prediction lengths

数据集	L_pred	FSDformer		Autoformer		Informer		LogTrans		Pyraformer		LST-Net
数据集	L_pred	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Electricity	96	0.180	0.288	0.206	0.321	0.324	0.423	0.262	0.358	0.386	0.449	0.283	0.357
	192	0.205	0.315	0.250	0.356	0.355	0.438	0.271	0.371	0.386	0.443	0.329	0.385
	336	0.223	0.332	0.288	0.375	0.396	0.468	0.282	0.382	0.378	0.443	0.357	0.391
	720	0.262	0.364	0.282	0.384	0.792	0.687	0.285	0.376	0.376	0.445	0.442	0.442
Traffic	96	0.597	0.383	0.658	0.411	0.797	0.452	0.685	0.384	2.085	0.468	1.107	0.668
	192	0.634	0.403	0.652	0.412	0.946	0.535	0.688	0.392	0.867	0.467	1.159	0.712
	336	0.647	0.404	0.652	0.407	1.382	0.760	0.735	0.408	0.869	0.469	1.220	0.733
	720	0.677	0.428	0.684	0.423	1.152	0.649	0.717	0.397	0.881	0.473	1.482	0.807

Tab. 3 Prediction results of multivariate time-series with different input lengths on Electricity dataset

$L i n$	FSDformer				Autoformer				Informer
	$L p r e d = 24$		$L p r e d = 720$		$L p r e d = 24$		$L p r e d = 720$		$L p r e d = 24$		$L p r e d = 720$
	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
72	0.157	0.274	0.262	0.367	0.165	0.285	0.279	0.376	0.285	0.385	0.370	0.439
96	0.162	0.278	0.262	0.369	0.174	0.291	0.282	0.384	0.299	0.394	0.389	0.454
120	0.146	0.262	0.252	0.359	0.167	0.288	0.260	0.363	0.337	0.419	0.394	0.456
144	0.144	0.258	0.250	0.360	0.168	0.287	0.264	0.366	0.391	0.445	0.400	0.463
168	0.137	0.251	0.250	0.357	0.161	0.279	0.239	0.348	0.318	0.410	0.396	0.458
192	0.129	0.242	0.232	0.342	0.165	0.284	0.265	0.368	0.307	0.399	0.414	0.472
336	0.126	0.241	0.216	0.326	0.170	0.291	0.247	0.351	0.354	0.440	0.424	0.477
504	0.123	0.237	0.225	0.332	0.180	0.300	0.249	0.355	0.358	0.443	0.423	0.475
672	0.124	0.237	0.229	0.338	0.188	0.304	0.273	0.371	0.327	0.416	0.378	0.443
720	0.121	0.234	0.220	0.327	0.186	0.302	0.254	0.356	0.355	0.432	0.401	0.456
960	0.121	0.235	0.221	0.330	0.194	0.309	0.275	0.368	0.411	0.457	0.400	0.451

Tab. 3 Prediction results of multivariate time-series with different input lengths on Electricity dataset

$L i n$	FSDformer				Autoformer				Informer
	$L p r e d = 24$		$L p r e d = 720$		$L p r e d = 24$		$L p r e d = 720$		$L p r e d = 24$		$L p r e d = 720$
	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
72	0.157	0.274	0.262	0.367	0.165	0.285	0.279	0.376	0.285	0.385	0.370	0.439
96	0.162	0.278	0.262	0.369	0.174	0.291	0.282	0.384	0.299	0.394	0.389	0.454
120	0.146	0.262	0.252	0.359	0.167	0.288	0.260	0.363	0.337	0.419	0.394	0.456
144	0.144	0.258	0.250	0.360	0.168	0.287	0.264	0.366	0.391	0.445	0.400	0.463
168	0.137	0.251	0.250	0.357	0.161	0.279	0.239	0.348	0.318	0.410	0.396	0.458
192	0.129	0.242	0.232	0.342	0.165	0.284	0.265	0.368	0.307	0.399	0.414	0.472
336	0.126	0.241	0.216	0.326	0.170	0.291	0.247	0.351	0.354	0.440	0.424	0.477
504	0.123	0.237	0.225	0.332	0.180	0.300	0.249	0.355	0.358	0.443	0.423	0.475
672	0.124	0.237	0.229	0.338	0.188	0.304	0.273	0.371	0.327	0.416	0.378	0.443
720	0.121	0.234	0.220	0.327	0.186	0.302	0.254	0.356	0.355	0.432	0.401	0.456
960	0.121	0.235	0.221	0.330	0.194	0.309	0.275	0.368	0.411	0.457	0.400	0.451

Fig. 9 Prediction error change of each model with different input lengths on Electricity dataset

Fig. 10 Loss changes with different Wres on Traffic dataset

Tab. 4 Model prediction results with different attention mechanisms on Electricity dataset

L_in	FSDformer		FSDformer-Full
L_in	MSE	MAE	MSE	MAE
72	0.157	0.274	0.176	0.293
96	0.162	0.278	0.169	0.288
168	0.146	0.262	0.149	0.266
192	0.144	0.258	0.140	0.258
336	0.137	0.251	0.139	0.257
504	0.129	0.242	0.135	0.252
672	0.126	0.241	0.132	0.247
720	0.123	0.237	0.127	0.242
960	0.124	0.237	0.132	0.249

Tab. 5 Prediction results before and after frequency sensitive dual-branch structure exchange on Electricity dataset

L_in	FSDformer		Exchange
L_in	MSE	MAE	MSE	MAE
72	0.157	0.274	0.306	0.406
96	0.162	0.278	0.295	0.399
168	0.146	0.262	0.265	0.376
192	0.144	0.258	0.266	0.375
336	0.137	0.251	0.292	0.393
504	0.129	0.242	0.286	0.389
672	0.126	0.241	0.253	0.364
720	0.123	0.237	0.244	0.362
960	0.124	0.237	0.267	0.372

Fig. 11 Comparison of training time on Electricity dataset

Fig. 12 Comparison of prediction accuracy under different EncoderLayer numbers on Electricity dataset

References 25

1	李大社，孙元威，阮俊虎. 基于GOSSA和HMM的时间序列预测算法［J］. 电子学报， 2023， 51（9）：2492-2503.
	LI D S， SUN Y W， RUAN J H. Time series prediction algorithm based on GOSSA and HMM ［J］. Acta Electronica Sinica， 2023， 51（9）：2492-2503.
2	SHIH S Y， SUN F K， LEE H Y. Temporal pattern attention for multivariate time series forecasting ［J］. Machine Learning， 2019， 108（8/9）： 1421-1441.
3	WU J， XU K， CHEN X， et al. Price graphs： utilizing the structural information of financial time series for stock prediction ［J］. Information Sciences， 2022， 588： 405-424.
4	PEREIRA D F， LOPES F D C， WATANABE E H. Nonlinear model predictive control for the energy management of fuel cell hybrid electric vehicles in real time ［J］. IEEE Transactions on Industrial Electronics， 2021， 68（4）： 3213-3223.
5	SUN P， ALJERI N， BOUKERCHE A. Machine learning-based models for real-time traffic flow prediction in vehicular networks［J］. IEEE Network， 2020， 34（3）： 178-185.
6	KAREVAN Z， SUYKENS J A K. Transductive LSTM for time-series prediction： an application to weather forecasting ［J］. Neural Networks， 2020， 125： 1-9.
7	LIU Y， GONG C， YANG L， et al. DSTP-RNN： a dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction ［J］. Expert Systems with Applications， 2020， 143： No.113082.
8	SAGHEER A， KOTB M. Time series forecasting of petroleum production using deep LSTM recurrent networks ［J］. Neurocomputing， 2019， 323： 203-213.
9	ZHOU H， ZHANG S， PENG J， et al. Informer： beyond efficient Transformer for long sequence time-series forecasting ［C］// Proceedings of the 35th AAAI Conference on Artificial Intelligence. Palo Alto： AAAI Press， 2021： 11106-11115.
10	ILHAN F， KARAAHMETOGLU O， BALABAN I， et al. Markovian RNN： an adaptive time series prediction network with HMM-based switching for nonstationary environments ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2023， 34（2）： 715-728.
11	XU X， YONEDA M. Multitask air-quality prediction based on LSTM-autoencoder model ［J］. IEEE Transactions on Cybernetics， 2021， 51（5）： 2577-2586.
12	LI X， MA X， XIAO F， et al. Time-series production forecasting method based on the integration of Bidirectional Gated Recurrent Unit （Bi-GRU） network and Sparrow Search Algorithm （SSA）［J］. Journal of Petroleum Science and Engineering， 2022， 208（Pt A）： No.109309.
13	LAI G， CHANG W C， YANG Y， et al. Modeling long- and short-term temporal patterns with deep neural networks ［C］// Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. New York： ACM， 2018： 95-104.
14	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 6000-6010.
15	LIU S， YU H， LIAO C， et al. Pyraformer： low-complexity pyramidal attention for long-range time series modeling and forecasting ［EB/OL］. （2023-02-14）［2023-05-23］. .
16	ZHOU T， MA Z， WEN Q， et al. FEDformer： frequency enhanced decomposed Transformer for long-term series forecasting ［C］// Proceedings of the 39th International Conference on Machine Learning. New York： JMLR.org， 2022： 27268-27286.
17	TAY Y， DEHGHANI M， ABNAR S， et al. Long range arena： a benchmark for efficient Transformers ［EB/OL］. （2020-11-08）［2023-03-22］. .
18	XIONG T， LI C， BAO Y. Seasonal forecasting of agricultural commodity price using a hybrid STL and ELM method： evidence from the vegetable market in China ［J］. Neurocomputing， 2018， 275： 2831-2844.
19	WEN Q， ZHANG Z， LI Y， et al. Fast RobustSTL： efficient and robust seasonal-trend decomposition for time series with complex patterns ［C］// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2020： 2203-2213.
20	GUO L， FANG W， ZHAO Q， et al. The hybrid PROPHET-SVR approach for forecasting product time series demand with seasonality ［J］. Computers and Industrial Engineering， 2021， 161： No.107598.
21	SEN R， YU H F， DHILLON I. Think globally， act locally： a deep neural network approach to high-dimensional time series forecasting ［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2019： No.435.
22	WU H， XU J， WANG J， et al. Autoformer： decomposition Transformers with auto-correlation for long-term series forecasting［EB/OL］. ［2023-04-14］. .
23	金苍宏，董腾然，陈天翼，等. 融合序列分解与时空卷积的时序预测算法［J］. 电子学报， 2021， 49（2）： 233-238.
	JIN C H， DONG T R， CHEN T Y， et al. Spatio-temporal convolutional forecasting based on time-series decomposition strategy ［J］. Acta Electronica Sinica， 2021， 49（2）： 233-238.
24	ASADI R， REGAN A C. A spatio-temporal decomposition based deep neural network for time series forecasting ［J］. Applied Soft Computing， 2020， 87： No.105963.
25	夏进，王正群，朱世明. 基于时间序列分解的交通流量预测模型［J］. 计算机应用， 2023， 43（4）： 1129-1135.
	XIA J， WANG Z Q， ZHU S M. Traffic flow prediction model based on time series decomposition ［J］. Journal of Computer Applications， 2023， 43（4）： 1129-1135.

[1]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[2]	Xin YANG, Xueni CHEN, Chunjiang WU, Shijie ZHOU. Short-term traffic flow prediction of urban highway based on variant residual model and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2947-2951.
[3]	Jinjin LI, Guoming SANG, Yijia ZHANG. Multi-domain fake news detection model enhanced by APK-CNN and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2674-2682.
[4]	Jieru JIA, Jianchao YANG, Shuorui ZHANG, Tao YAN, Bin CHEN. Unsupervised person re-identification based on self-distilled vision Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2893-2902.
[5]	Jiepo FANG, Chongben TAO. Hybrid internet of vehicles intrusion detection system for zero-day attacks [J]. Journal of Computer Applications, 2024, 44(9): 2763-2769.
[6]	Yuwei DING, Hongbo SHI, Jie LI, Min LIANG. Image denoising network based on local and global feature decoupling [J]. Journal of Computer Applications, 2024, 44(8): 2571-2579.
[7]	Kaili DENG, Weibo WEI, Zhenkuan PAN. Industrial defect detection method with improved masked autoencoder [J]. Journal of Computer Applications, 2024, 44(8): 2595-2603.
[8]	Fan YANG, Yao ZOU, Mingzhi ZHU, Zhenwei MA, Dawei CHENG, Changjun JIANG. Credit card fraud detection model based on graph attention Transformation neural network [J]. Journal of Computer Applications, 2024, 44(8): 2634-2642.
[9]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[10]	Junfeng SHEN, Xingchen ZHOU, Can TANG. Dual-channel sentiment analysis model based on improved prompt learning method [J]. Journal of Computer Applications, 2024, 44(6): 1796-1806.
[11]	Xiting LYU, Jinghua ZHAO, Haiying RONG, Jiale ZHAO. Information diffusion prediction model based on Transformer and relational graph convolutional network [J]. Journal of Computer Applications, 2024, 44(6): 1760-1766.
[12]	Xun YAO, Zhongzheng QIN, Jie YANG. Generative label adversarial text classification model [J]. Journal of Computer Applications, 2024, 44(6): 1781-1785.
[13]	Mengyuan HUANG, Kan CHANG, Mingyang LING, Xinjie WEI, Tuanfa QIN. Progressive enhancement algorithm for low-light images based on layer guidance [J]. Journal of Computer Applications, 2024, 44(6): 1911-1919.
[14]	Shibin LI, Jun GONG, Shengjun TANG. Semi-supervised heterophilic graph representation learning model based on Graph Transformer [J]. Journal of Computer Applications, 2024, 44(6): 1816-1823.
[15]	Ziwen SUN, Lizhi QIAN, Chuandong YANG, Yibo GAO, Qingyang LU, Guanglin YUAN. Survey of visual object tracking methods based on Transformer [J]. Journal of Computer Applications, 2024, 44(5): 1644-1654.

Multivariate long-term series forecasting method with DFT-based frequency-sensitive dual-branch Transformer

基于DFT的频率敏感双分支Transformer多变量长时间序列预测方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 25

Related Articles 15

Recommended Articles

Metrics