Time series prediction model based on multimodal information fusion

doi:10.11772/j.issn.1001-9081.2021061053

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (8): 2326-2332.DOI: 10.11772/j.issn.1001-9081.2021061053

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Time series prediction model based on multimodal information fusion

Minghui WU¹(), Guangjie ZHANG¹^,², Canghong JIN¹

^1.School of Computer and Computing Science，Zhejiang University City College，Hangzhou Zhejiang 310015，China
^2.College of Computer Science and Technology，Zhejiang University，Hangzhou Zhejiang 310027，China

Received:2021-06-24 Revised:2021-08-24 Accepted:2021-09-28 Online:2021-10-18 Published:2022-08-10
Contact: Minghui WU
About author:WU Minghui， born in 1976， Ph. D.， professor. His research interests include artificial intelligence， software engineering， mobile computing.
ZHANG Guangjie， born in 1996， M. S. Her research interests include natural language processing， machine learning.
JIN Canghong， born in 1982， Ph. D.， associate professor. His research interests include data mining， artificial intelligence.
Supported by:
Zhejiang Provincial Key Research and Development Program(2021C01164)

基于多模态信息融合的时间序列预测模型

吴明晖¹(), 张广洁¹^,², 金苍宏¹

^1.浙大城市学院计算机与计算科学学院，杭州 310015
^2.浙江大学计算机科学与技术学院，杭州 310027

通讯作者: 吴明晖
作者简介:吴明晖（1976—），男，江西南昌人，教授，博士，CCF会员，主要研究方向：人工智能、软件工程、移动计算；
张广洁（1996—），女，安徽淮南人，硕士，主要研究方向：自然语言处理、机器学习；
金苍宏（1982—），男，浙江绍兴人，副教授，博士，CCF会员，主要研究方向：数据挖掘、人工智能。
基金资助:
浙江省重点研发计划项目(2021C01164)

Abstract

Abstract:

Aiming at the problem that traditional single factor methods cannot make full use of the relevant information of time series and has the poor accuracy and reliability of time series prediction， a time series prediction model based on multimodal information fusion，namely Skip-Fusion， was proposed to fuse the text data and numerical data in multimodal data. Firstly， different types of text data were encoded by pre-trained Bidirectional Encoder Representations from Transformers （BERT） model and one-hot encoding. Then， the single vector representation of the multi-text feature fusion was obtained by using the pre-trained model based on global attention mechanism. After that， the obtained single vector representation was aligned with the numerical data in time order. Finally， the fusion of text and numerical features was realized through Temporal Convolutional Network （TCN） model， and the shallow and deep features of multimodal data were fused again through skip connection. Experiments were carried out on the dataset of stock price series， Skip-Fusion model obtains the results of 0.492 and 0.930 on the Root Mean Square Error （RMSE） and daily Return （R） respectively， which are better than the results of the existing single-modal and multimodal fusion models. Experimental results show that Skip-Fusion model obtains the goodness of fit of 0.955 on the R-squared， indicating that Skip-Fusion model can effectively carry out multimodal information fusion and has high accuracy and reliability of prediction.

Key words: global attention mechanism, skip connection, multimodal fusion, time series prediction, stock price prediction

摘要：

针对传统单因子模型无法充分利用时间序列相关信息，以及这些模型对时间序列预测准确性和可靠性较差的问题，提出一种基于多模态信息融合的时间序列预测模型——Skip-Fusion对多模态数据中的文本数据和数值数据进行融合。首先利用BERT（Bidirectional Encoder Representations from Transformers）预训练模型和独热编码对不同类别的文本数据进行编码表示；再使用基于全局注意力机制的预训练模型获得多文本特征融合的单一向量表示；然后将得到的单一向量表示与数值数据按时间顺序对齐；最后通过时间卷积网络（TCN）模型实现文本和数值特征的融合，并通过跳跃连接完成多模态数据的浅层和深层特征的再次融合。在股票价格序列的数据集上进行实验，Skip-Fusion模型的均方根误差（RMSE）和日收益（R）分别为0.492和0.930，均优于现有的单模态模型和多模态融合模型的结果，同时在可决系数（R-Squared）上取得了0.955的拟合优度。实验结果表明，Skip-Fusion模型能够有效进行多模态信息融合并具有较高的预测准确性和可靠性。

关键词: 全局注意力机制, 跳跃连接, 多模态融合, 时间序列预测, 股票价格预测

CLC Number:

TP391.4

Minghui WU, Guangjie ZHANG, Canghong JIN. Time series prediction model based on multimodal information fusion[J]. Journal of Computer Applications, 2022, 42(8): 2326-2332.

吴明晖, 张广洁, 金苍宏. 基于多模态信息融合的时间序列预测模型[J]. 《计算机应用》唯一官方网站, 2022, 42(8): 2326-2332.

Figures/Tables 7

Fig. 1 Framework of the proposed model

Fig. 2 Structure of global attention mechanism

Fig. 3 Skip connection

Tab. 1 Examples of research report data

标题	行业	评级	日期	研报机构名称
建筑材料行业：年内最难时候已过，水泥价格韧性可期	水泥建材	持有	2020-03-25	广发证券股份有限公司
医药行业周报：海外疫情升温，医药行业内部调整	医药制造	中性	2020-03-25	渤海证券股份有限公司
新能源汽车：北京计划松绑新能源指标，逆周期刺激政策浮出水面	汽车行业	买入	2020-03-25	广发证券股份有限公司
汽车行业跟踪报告：海外疫情影响，国内汽车政策力度有望加强	汽车行业	推荐	2020-03-25	华创证券有限责任公司

Tab. 2 Parameter setting of multimodal fusion model

参数	值
TCN层数	1
TCN通道大小	600
TCN卷积核大小	4
TCN空洞卷积空洞大小	2
词级别向量维度	28
段落级别向量维度	100
历史序列长度	30
预测序列长度	5
批次大小	32
初始学习率	0.001

Tab. 3 Experimental results of single-modal and multimodal fusion models

模型		RMSE	R-Squared	R
单模态	Naïve	3.059	0.996	0.501
	MA	3.883	0.994	0.181
	EMA	4.366	0.992	0.163
	ARIMA	4.126	0.993	-0.302
	GRU	1.059	0.536	0.817
	TCN	0.708	0.821	0.888
多模态	Concat	0.660	0.873	0.910
	Hadamard	0.780	0.827	0.880
	MLB	0.716	0.856	0.895
	MFB	0.754	0.852	0.891
	Skip-Fusion	0.492	0.955	0.930

Tab. 4 Time complexity analysis

模型		时间复杂度
单模态	Naïve	$O (1)$
	MA	$O (T)$
	EMA	$O (T)$
	ARIMA	$O (T)$
	GRU	$O (T × H G R U)$
	TCN	$O (T × H T C N)$
多模态	Concat	$O (T × H F + T × H T C N)$
	Hadamard	$O (T × H F + T × H T C N)$
	MLB	$O (T × H M + T × H T C N)$
	MFB	$O (T × K × H M + T × H T C N)$
	Skip-Fusion	$O (T × H F + T × H T C N)$

Tab. 4 Time complexity analysis

模型		时间复杂度
单模态	Naïve	$O (1)$
	MA	$O (T)$
	EMA	$O (T)$
	ARIMA	$O (T)$
	GRU	$O (T × H G R U)$
	TCN	$O (T × H T C N)$
多模态	Concat	$O (T × H F + T × H T C N)$
	Hadamard	$O (T × H F + T × H T C N)$
	MLB	$O (T × H M + T × H T C N)$
	MFB	$O (T × K × H M + T × H T C N)$
	Skip-Fusion	$O (T × H F + T × H T C N)$

References 15

1	张碧琼，李越.汇率对中国股票市场的影响是否存在：从自回归分布滞后模型（ARDL-ecm）得到的证明［J］.金融研究， 2002（7）： 26-35.
	ZHANG B Q， LI Y. Do the exchange rates matter for Chinese stock market： evidence from ARDL-ecm estimation［J］. Journal of Financial Research， 2002（7）： 26-35.
2	CONTRERAS J， ESPINOLA R， NOGALES F J， et al. ARIMA models to predict next-day electricity prices［J］. IEEE Transactions on Power Systems， 2003， 18（3）： 1014-1020. 10.1109/tpwrs.2002.804943
3	GAO T W， CHAI Y T. Improving stock closing price prediction using recurrent neural network and technical indicators［J］. Neural Computation， 2018， 30（10）： 2833-2854. 10.1162/neco_a_01124
4	姚小强，侯志森.基于树结构长短期记忆神经网络的金融时间序列预测［J］.计算机应用， 2018， 38（11）： 3336-3341. 10.11772/j.issn.1001-9081.2018040742
	YAO X Q， HOU Z S. Financial time series prediction by long short term memory neural network with tree structure［J］. Journal of Computer Applications， 2018， 38（11）： 3336-3341. 10.11772/j.issn.1001-9081.2018040742
5	张栗粽，王谨平，刘贵松，等.面向金融数据的神经网络时间序列预测模型［J］.计算机应用研究， 2018， 35（9）： 2632-2637. 10.3969/j.issn.1001-3695.2018.09.017
	ZHANG L Z， WANG J P， LIU G S， et al. Neural network time series prediction model for financial data［J］. Application Research of Computers， 2018， 35（9）： 2632-2637. 10.3969/j.issn.1001-3695.2018.09.017
6	GAO P， JIANG Z K， YOU H X， et al. Dynamic fusion with intra- and inter-modality attention flow for visual question answering ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 6632-6641. 10.1109/cvpr.2019.00680
7	孔震，张华鲁，岳圣凯，等.基于时域卷积网络的多尺度双线性天气预测模型［J］.图学学报， 2020， 41（5）： 764-770.
	KONG Z， ZHANG H L， YUE S K， et al. Multi-scale and bilinear models based on temporal convolutional network［J］. Journal of Graphics， 2020， 41（5）： 764-770.
8	BAI S J， KOLTER J Z， KOLTUN V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling［EB/OL］. （2018-04-19）［2020-08-23］. .
9	KIM J H， ON K W， LIM W， et al. Hadamard product for low-rank bilinear pooling［EB/OL］. （2017-03-26）［2020-08-23］. .
10	YU Z， YU J， FAN J P， et al. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 1839-1848. 10.1109/iccv.2017.202
11	DEVLIN J， CHANG M W， LEE K， et al. BERT： pre-training of deep bidirectional transformers for language understanding ［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics： Human Language Technologies， Volume 1（Long and Short Papers）. Stroudsburg， PA： Association for Computational Linguistics， 2019： 4171-4186.
12	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 6000-6010. 10.1016/s0262-4079(17)32358-8
13	ZHANG X， SHEN F R， ZHAO J X， et al. Time series forecasting using GRU neural network with multi-lag after decomposition ［C］// Proceedings of the 2017 International Conference on Neural Information Processing， LNCS 10638. Cham： Springer， 2017： 523-532.
14	DENG S M， ZHANG N Y， ZHANG W， et al. Knowledge-driven stock trend prediction and explanation via temporal convolutional network ［C］// Proceedings of the 2019 World Wide Web Conference. New York： ACM， 2019： 678-685. 10.1145/3308560.3317701
15	FUKUI A， PARK D H， YANG D， et al. Multimodal compact bilinear pooling for visual question answering and visual grounding ［C］// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg， PA： Association for Computational Linguistics， 2016： 457-468. 10.18653/v1/d16-1044

[1]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[2]	Yu ZENG, Yang ZHANG, Shang ZENG, Maoli FU, Qixue HE, Linlong ZENG. Time series prediction algorithm based on multi-scale gated dilated convolutional network [J]. Journal of Computer Applications, 2024, 44(11): 3427-3434.
[3]	Cong GU, Qiqiang DUAN, Siyu REN. Polyp segmentation algorithm based on context-aware network [J]. Journal of Computer Applications, 2024, 44(11): 3617-3622.
[4]	Chunlei WANG, Xiao WANG, Kai LIU. Multimodal knowledge graph representation learning： a review [J]. Journal of Computer Applications, 2024, 44(1): 1-15.
[5]	Junhao LUO, Yan ZHU. Multi-dynamic aware network for unaligned multimodal language sequence sentiment analysis [J]. Journal of Computer Applications, 2024, 44(1): 79-85.
[6]	Mu LI, Yuheng YANG, Xizheng KE. Emotion recognition model based on hybrid-mel gama frequency cross-attention transformer modal [J]. Journal of Computer Applications, 2024, 44(1): 86-93.
[7]	Qiang ZHAO, Zhongqing WANG, Hongling WANG. Product summarization extraction model with multimodal information fusion [J]. Journal of Computer Applications, 2024, 44(1): 73-78.
[8]	Juntao CHEN, Ziqi ZHU. Image copy-move forgery detection based on multi-scale feature extraction and fusion [J]. Journal of Computer Applications, 2023, 43(9): 2919-2924.
[9]	Qiuyu ZHANG, Yukun WANG. Speech classification model based on improved Inception network [J]. Journal of Computer Applications, 2023, 43(3): 909-915.
[10]	Yizhen BI, Huan MA, Changqing ZHANG. Dynamic evaluation method for benefit of modality augmentation [J]. Journal of Computer Applications, 2023, 43(10): 3099-3106.
[11]	Xiaoyan LU, Yang XU, Wenhao YUAN. Multiscale dense fusion network for lung lesion image segmentation [J]. Journal of Computer Applications, 2023, 43(10): 3282-3289.
[12]	Yifei WANG, Lei YU, Fei TENG, Jiayu SONG, Yue YUAN. Resource load prediction model based on long-short time series feature fusion [J]. Journal of Computer Applications, 2022, 42(5): 1508-1515.
[13]	Hexuan HU, Huachao SUI, Qiang HU, Ye ZHANG, Zhenyun HU, Nengwu MA. Runoff forecast model based on graph attention network and dual-stage attention mechanism [J]. Journal of Computer Applications, 2022, 42(5): 1607-1615.
[14]	Yuxi LIU, Yuqi LIU, Zonglin ZHANG, Zhihua WEI, Ran MIAO. News recommendation model with deep feature fusion injecting attention mechanism [J]. Journal of Computer Applications, 2022, 42(2): 426-432.
[15]	LI Yangying, CHEN Zhijun, ZHANG Zihao, YOU Lan. Raw sugar demand forecasting model for sugar manufacturing enterprise based on modified Elman neural network [J]. Journal of Computer Applications, 2021, 41(7): 2113-2120.

Time series prediction model based on multimodal information fusion

基于多模态信息融合的时间序列预测模型

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 15

Related Articles 15

Recommended Articles

Metrics