Time series classification method based on multi-scale cross-attention fusion in time-frequency domain

doi:10.11772/j.issn.1001-9081.2023060731

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (6): 1842-1847.DOI: 10.11772/j.issn.1001-9081.2023060731

Special Issue: 数据科学与技术

• Data science and technology • Previous Articles Next Articles

Time series classification method based on multi-scale cross-attention fusion in time-frequency domain

Mei WANG¹^,², Xuesong SU¹, Jia LIU¹, Ruonan YIN²(), Shan HUANG¹^,³

^1.Shengli Oilfield Technical Testing Center，China Petroleum and Chemical Corporation，Dongying Shandong 257000，China
^2.School of Computer Science and Technology，China University of Petroleum （East China），Qingdao Shandong 266000，China
^3.College of Pipeline and Civil Engineering，China University of Petroleum （East China），Qingdao Shandong 266000，China

Received:2023-06-07 Revised:2023-09-08 Accepted:2023-09-14 Online:2023-09-27 Published:2024-06-10
Contact: Ruonan YIN
About author:WANG Mei， born in 1981， Ph. D. candidate， senior engineer. Her research interests include artificial intelligence， time series analysis and application.
SU Xuesong， born in 1989， Ph. D. His research interests include oilfield data mining， artificial intelligence.
LIU Jia， born in 1981， M. S.， senior engineer. His research interests include oilfield data mining.
HUANG Shan， born in 1989， senior engineer. Her research interests include standard informatization in oilfield.
Supported by:
SINOPEC Scientific Research Project(323016);Scientific Research Project of Shengli Oilfield Branch(YG2208)

时频域多尺度交叉注意力融合的时间序列分类方法

王美¹^,², 苏雪松¹, 刘佳¹, 殷若南²(), 黄珊¹^,³

^1.中国石油化工股份有限公司胜利油田技术检测中心, 山东东营 257000
^2.中国石油大学(华东) 计算机科学与技术学院, 山东青岛 266000
^3.中国石油大学(华东) 储运与建筑工程学院, 山东青岛 266000

通讯作者: 殷若南
作者简介:王美（1981—），女，山东威海人，高级工程师，博士研究生，主要研究方向：人工智能、时间序列分析及应用
苏雪松（1989—），男，山东东营人，博士，主要研究方向：油田数据挖掘、人工智能
刘佳（1981—），男，山东威海人，高级工程师，硕士，主要研究方向：油田数据挖掘
黄珊（1989—），女，河北沧州人，高级工程师，主要研究方向：石油领域标准信息化。
基金资助:
中国石化股份公司科研项目(323016);胜利油田分公司科研项目(YG2208)

Abstract

Abstract:

To address the problem of low classification accuracy caused by insufficient potential information interaction between time series subsequences， a time series classification method based on multi-scale cross-attention fusion in time-frequency domain called TFFormer （Time-Frequency Transformer） was proposed. First， time and frequency spectrums of the original time series were divided into subsequences with the same length respectively， and the point-value coupling problem was solved by adding positional embedding after linear projection. Then， the long-term time series dependency problem was solved because the model was made to focus on more important time series features by Improved Multi-Head self-Attention （IMHA） mechanism. Finally， a multi-scale Cross-Modality Attention （CMA） module was proposed to enhance the interaction between the time domain and frequency domain， so that the model could further mine the frequency information of the time series. The experimental results show that compared with Fully Convolutional Network （FCN）， the classification accuracy of the proposed method on Trace， StarLightCurves and UWaveGestureLibraryAll datasets increased by 0.3， 0.9 and 1.4 percentage points. It is proved that by enhancing the information interaction between time domain and frequency domain of the time series， the model convergence speed and classification accuracy can be improved.

Key words: time series, attention mechanism, positional embedding, deep neural network, multi-scale fusion

摘要：

针对时间序列子序列间的潜在信息交互不足导致分类准确率低的问题，提出时频域多尺度交叉注意力融合的时间序列分类方法TFFormer（Time-Frequency Transformer）。首先，将原始时间序列的时频域谱分别划分为等长子序列，经线性投影后加入位置信息解决时间序列的点值耦合问题；其次，通过改进的多头自注意力（IMHA）模块使模型关注更重要的序列特征，解决长时间序列的前后依赖问题；最后，构造多尺度时频域交叉注意力（CMA）模块增强时间序列在时域和频域之间的信息交互，使模型进一步挖掘序列的频域信息。实验结果表明，在Trace、StarLightCurves和UWaveGestureLibraryAll数据集上，相较于全卷积网络（FCN），所提方法的分类准确率分别提高了0.3、0.9和1.4个百分点，验证了通过增强时间序列时域和频域间的信息交互，可以提高模型收敛速度和分类精度。

关键词: 时间序列, 注意力机制, 位置编码, 深度神经网络, 多尺度融合

CLC Number:

TP399

Mei WANG, Xuesong SU, Jia LIU, Ruonan YIN, Shan HUANG. Time series classification method based on multi-scale cross-attention fusion in time-frequency domain[J]. Journal of Computer Applications, 2024, 44(6): 1842-1847.

王美, 苏雪松, 刘佳, 殷若南, 黄珊. 时频域多尺度交叉注意力融合的时间序列分类方法[J]. 《计算机应用》唯一官方网站, 2024, 44(6): 1842-1847.

Figures/Tables 10

References 20

1	ISMAIL FAWAZ H， FORESTIER G， WEBER J， et al. Deep learning for time series classification： a review［J］. Data Mining and Knowledge Discovery， 2019， 33（4）： 917-963.
2	NWEKE H F， WAH T Y， AL-GARADI M， et al. Deep learning algorithms for human activity recognition using mobile and wearable sensor networks： state of the art and research challenges ［J］. Expert Systems with Applications， 2018， 105： 233-261.
3	NWE T L， DAT T H， MA B. Convolutional neural network with multi-task learning scheme for acoustic scene classification［C］// Proceedings of the 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference. Piscataway： IEEE， 2017： 1347-1350.
4	WANG J， LIU P， SHE M F H， et al. Bag-of-words representation for biomedical time series classification［J］. Biomedical Signal Processing and Control， 2013， 8（6）： 634-644.
5	LINES J， BAGNALL A. Time series classification with ensembles of elastic distance measures［J］. Data Mining and Knowledge Discovery， 2015， 29（3）： 565-592.
6	JI C， ZOU X， HU Y， et al. XG-SF： an XGBoost classifier based on shapelet features for time series classification［J］. Procedia Computer Science， 2019， 147： 24-28.
7	张雅雯，王志海，刘海洋，等. 基于多尺度残差 FCN 的时间序列分类算法［J］. 软件学报， 2022， 33（2）： 555-570.
	ZHANG Y W， WANG Z H， LIU H Y， et al. Time series classification algorithm based on multi-scale residual full convolutional neural network ［J］. Journal of Software， 2022， 33（2）： 555-570.
8	王俊陆，李素，纪婉婷，等. 基于 Gram 矩阵的 T-CNN 时间序列分类方法［J］. 浙江大学学报（工学版）， 2023， 57（2）： 267-276.
	WANG J L， LI S， JI W T， et al. T-CNN time series classification method based on Gram matrix［J］. Journal of Zhejiang University （Engineering Science）， 2023， 57（2）： 267-276.
9	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need ［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 6000-6010.
10	CHUA L O， ROSKA T. The CNN paradigm［J］. IEEE Transactions on Circuits and Systems Ⅰ： Fundamental Theory and Applications， 1993， 40（3）： 147-156.
11	FABBRI M， MORO G. Dow Jones trading with deep learning： the unreasonable effectiveness of recurrent neural networks［C］// Proceedings of the 7th International Conference on Data Science， Technology and Applications. Setubal， Portugal： SCITEPRESS， 2018： 142-153.
12	HU J， SHEN L， SUN G. Squeeze-and-excitation networks ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7132-7141.
13	LIN M， CHEN Q， YAN S. Network in network ［EB/OL］. ［2023-06-05］. .
14	KINGMA D P， BA J. Adam： a method for stochastic optimization ［EB/OL］. ［2023-06-05］. .
15	LIAW A， WIENER M. Classification and regression by randomForest ［J］. R News， 2002， 2/3： 18-22.
16	CHEN T， HE T， BENESTY M， et al. XGboost： extreme gradient boosting ［EB/OL］. ［2023-05-02］. .
17	RAMCHOUN H， IDRISSI M A J， GHANOU Y， et al. Multilayer perceptron： architecture optimization and training［J］. International Journal of Interactive Multimedia and Artificial Intelligence， 2016， 4（1）： 26-30.
18	HOCHREITER S， SCHMIDHUBER J. Long short-term memory［J］. Neural Computation， 1997， 9（8）： 1735-1780.
19	YAO H， TANG X， WEI H， et al. Modeling spatial-temporal dynamics for traffic prediction ［EB/OL］. ［2023-06-05］. .
20	LONG J， SHELHAMER E， DARRELL T. Fully convolutional networks for semantic segmentation ［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 3431-3440.

数据集	长度	训练集大小	测试集大小	类别数
Plane	144	104	104	7
Trace	275	99	99	4
MixedShapesRegularTrain	1 024	499	2 424	5
StarLightCurves	1 024	999	8 235	3
UWaveGestureLibraryAll	945	896	3 582	8

数据集	长度	训练集大小	测试集大小	类别数
Plane	144	104	104	7
Trace	275	99	99	4
MixedShapesRegularTrain	1 024	499	2 424	5
StarLightCurves	1 024	999	8 235	3
UWaveGestureLibraryAll	945	896	3 582	8

方法	不同数据集的分类准确率
方法	Plane	Trace	Mixed-Shapes- RegularTrain	StarLight- Curves	UWave- Gesture- LibraryAll
RF	0.942	0.980	0.884	0.950	0.893
XGBoost	0.952	0.939	0.886	0.957	0.901
MLP	0.981	0.920	0.876	0.925	0.901
LSTM	0.983	0.962	0.890	0.902	0.907
LSTM+CNN	0.989	1.000	0.885	0.949	0.911
FCN	1.000	0.997	0.939	0.967	0.909
Transformer	0.942	0.904	0.910	0.940	0.899
TFFormer	1.000	1.000	0.937	0.976	0.923

方法	不同数据集的分类准确率
方法	Plane	Trace	Mixed-Shapes- RegularTrain	StarLight- Curves	UWave- Gesture- LibraryAll
RF	0.942	0.980	0.884	0.950	0.893
XGBoost	0.952	0.939	0.886	0.957	0.901
MLP	0.981	0.920	0.876	0.925	0.901
LSTM	0.983	0.962	0.890	0.902	0.907
LSTM+CNN	0.989	1.000	0.885	0.949	0.911
FCN	1.000	0.997	0.939	0.967	0.909
Transformer	0.942	0.904	0.910	0.940	0.899
TFFormer	1.000	1.000	0.937	0.976	0.923

方法	不同数据集的分类准确率
方法	Plane	Trace	Mixed- Shapes- RegularTrain	StarLight- Curves	UWave- Gesture- LibraryAll
TO	0.921	0.907	0.830	0.900	0.876
AbsPE+CMA+SAF	0.987	0.985	0.910	0.956	0.919
LPE+CMA	0.966	0.959	0.898	0.934	0.900
LPE+SAF	0.971	0.956	0.902	0.958	0.898
本文方法（Hilbert）	1.000	0.998	0.940	0.977	0.920
本文方法	1.000	1.000	0.937	0.976	0.923

Time series classification method based on multi-scale cross-attention fusion in time-frequency domain

时频域多尺度交叉注意力融合的时间序列分类方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 20

Related Articles 15

Recommended Articles

Metrics

序列长度	序列数	准确率	序列长度	序列数	准确率
4	4	1.000	16	2	0.989
4	8	0.985	16	4	0.700
8	4	0.954	32	1	0.980
8	8	0.739	32	2	0.689

[1]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[2]	Qinzhuang ZHAO, Hongye TAN. Time series causal inference method based on adaptive threshold learning [J]. Journal of Computer Applications, 2024, 44(9): 2660-2666.
[3]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[4]	Lilin FAN, Fukang CAO, Wanting WANG, Kai YANG, Zhaoyu SONG. Intermittent demand forecasting method based on adaptive matching of demand patterns [J]. Journal of Computer Applications, 2024, 44(9): 2747-2755.
[5]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[6]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[7]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[8]	Rui SHI, Yong LI, Yanhan ZHU. Adversarial sample attack algorithm of modulation signal based on equalization of feature gradient [J]. Journal of Computer Applications, 2024, 44(8): 2521-2527.
[9]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[10]	Chenqian LI, Jun LIU. Ultrasound carotid plaque segmentation method based on semi-supervision and multi-scale cascaded attention [J]. Journal of Computer Applications, 2024, 44(8): 2604-2610.
[11]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[12]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[13]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[14]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[15]	Wenliang WEI, Yangping WANG, Biao YUE, Anzheng WANG, Zhe ZHANG. Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention [J]. Journal of Computer Applications, 2024, 44(7): 2183-2191.