EEG decoding model based on dual-branch representation fusion and cross-modal alignment

doi:10.11772/j.issn.1001-9081.2025081016

Abstract

Abstract:

Concerning the problem that the existing text-oriented ElectroEncephaloGraphy （EEG） decoding methods focus on global modeling while neglecting inter-channel local correlations and failing to align EEG and text representations precisely， an EEG decoding model based on dual-branch representation fusion and cross-modal alignment was proposed. In the model， a parallel dual-branch architecture was adopted to obtain high-quality representations of EEG signals： in the spatio-temporal branch， signal short-term dependencies， neighboring-channel spatial correlations， and cross-step long-range relationships were captured layer by layer through Bidirectional Long Short-Term Memory （Bi-LSTM） network， depthwise separable convolution， and gated axial self-attention mechanism； in the context fusion branch， based on a multi-layer Transformer encoder， cross-attention was used to fuse the two representations for complementary integration. Besides， to reduce semantic discrepancies between modalities， a joint loss combining triplet loss and covariance alignment was introduced， thereby constraining vector alignment by the geometric distance and second-order statistical properties between paired EEG and text representations. Experimental results on the ZuCo dataset show that the proposed model improves BLEU-1 （BiLingual Evaluation Understudy-1） by about 1.16 percentage points compared with the baseline EEG2Text， demonstrating its effectiveness in text-oriented EEG decoding tasks.

Key words: ElectroEncephaloGraphy (EEG), deep neural network, feature fusion, modality discrepancy, text generation

摘要：

针对现有面向文本的脑电信号（EEG）解码方法偏重全局建模，忽视通道间局部关联且未实现EEG与文本表征精确对齐的问题，提出一种基于双分支表征融合与跨模态对齐的EEG解码模型。采用并行双分支架构获取EEG信号的高质量表征：时空结构建模分支通过双向长短期记忆（Bi-LSTM）网络、深度可分离卷积与门控轴向自注意力机制分层捕获信号的短时依赖、邻近通道空间相关性与跨步长程关系；上下文融合分支则基于多层Transformer编码器，通过交叉注意力融合两路表征以互补整合。为了弱化模态间的语义差异，引入三元组损失与协方差对齐的联合损失，分别从EEG与文本表征样本对间的几何距离和二阶统计特性两方面约束向量对齐。实验结果表明，在ZuCo数据集上，所提模型的BLEU-1（BiLingual Evaluation Understudy-1）相较于主流基线EEG2Text提升了1.16个百分点，验证了它在面向文本的EEG解码任务中的有效性。

关键词: 脑电信号, 深度神经网络, 特征融合, 模态差异性, 文本生成

CLC Number:

TP18

Xiaocui XU, Bo LI, Yutong ZOU. EEG decoding model based on dual-branch representation fusion and cross-modal alignment[J]. Journal of Computer Applications, 2026, 46(9): 3035-3042.

徐晓翠, 李波, 邹宇童. 基于双分支表征融合与跨模态对齐的脑电信号解码模型[J]. 《计算机应用》唯一官方网站, 2026, 46(9): 3035-3042.

Figures/Tables 9

Fig. 1 Architecture of proposed model

Tab. 1 Detailed information of ZuCo datasets

数据集	受试者数	唯一句子数	训练样本数	测试样本数
SR v1.0	12	400	3 391	418
NR v1.0	12	300	2 406	321
NR v2.0	18	349	4 456	601

Tab. 2 Model network setting

序号	模块	参数设置
1	Bi-LSTM	Insize=840， bidirectional， layers=1， hiddensize=420， outsize=1 024，激活=GELU
2	深度可分离卷积块	Insize=840， layers=3， outsize=840； depthwise： kernel=3，stride=1，pad=1； Pointwise： kernel_size=1；outsize=840；激活=ReLU
3	门控轴向自注意力	Insize=840， layers=6， len=56， heads=8， ffn_dim=2 048；时间轴MHSA dim=840；通道轴MHSA dim=56；门控激活=Sigmoid
4	Transformer 编码器	Insize=840， layers=6， d_model=840， heads=8， ffndim=2 048，outsize=840； projection： insize=840， outsize=1 024
5	交叉注意力层	Insize=1 024， embed_dim=1 024， num_heads=8， outsize=1 024
6	TextEncoder	Model=BART-Large， outsize=1 024

Tab. 3 Information of baseline model

基线ID	EEG编码器	预训练语言模型
EEG2Text^［15］	MTE	BART-Large^［16］
EEG2Text-B1	MTE	BART-Base
EEG2Text-B2	MTE	T5-base^［32］
EEG2Text-B3	MTE	Pegasus-Large^［33］
EEG2Text-B4	MTE	Pegasus-Sum

Tab. 4 Experiment results of different models

模型	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-p	ROUGE-r	ROUGE-f
EEG2Text	40.12	23.18	12.61	6.80	28.84	31.69	30.10
EEG2Text-B1	39.97	22.04	11.51	6.43	30.30	29.31	29.73
EEG2Text-B2	30.42	16.38	8.75	4.56	13.80	20.99	15.30
EEG2Text-B3	39.74	22.43	12.66	6.75	30.30	30.12	30.09
EEG2Text-B4	38.98	21.36	11.81	6.38	29.69	29.02	29.24
本文模型	41.28	24.14	13.88	8.01	32.31	29.90	31.01

Tab. 5 Ablation experiment results

模型	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-p	ROUGE-r	ROUGE-f
本文模型	41.28	24.14	13.88	8.01	32.31	29.90	31.01
消融1	40.35	23.14	12.94	7.28	31.95	29.19	30.47
消融2	40.00	22.60	12.11	6.37	31.77	29.19	30.38
消融3	40.38	23.21	12.74	6.92	32.54	29.44	30.87

Tab. 6 Impact of weights of joint loss on model performance

$λ 1$	$λ 2$	BLEU-1/%	ROUGE-f/%
1.0	1.0	41.28	31.01
1.0	0.0	40.52	30.80
0.0	1.0	39.73	30.52
2.0	1.0	40.43	30.40
1.5	1.0	40.91	30.93
1.2	0.8	41.13	31.04

Tab. 6 Impact of weights of joint loss on model performance

$λ 1$	$λ 2$	BLEU-1/%	ROUGE-f/%
1.0	1.0	41.28	31.01
1.0	0.0	40.52	30.80
0.0	1.0	39.73	30.52
2.0	1.0	40.43	30.40
1.5	1.0	40.91	30.93
1.2	0.8	41.13	31.04

Fig. 2 EEG-Text representation similarity histograms

Tab. 7 Case study

示例	类型	内容
1	目标句	While attending a motorcycle race， he met a local girlnamedColumba Garnica Gallo， whom he eventually married.
1	生成句	in the college rally in he met a young womannamed Marya，ett，o， who he later married.
2	目标句	Jeb Bushwas born inMidland， Texas， where his father was running an oil drilling company.
2	生成句	Bush， born intheway， Texas， and he fatherwas a a insurance company company.
3	目标句	Their three children， now grown， are George P. （recently married）， John E. "Jeb" Bush Jr. and Noelle.
3	生成句	first children， all grown， are all，. PGeorgelymarried）， George （.（Jackock"（（.，（ Johnelle （
4	目标句	Bush attended the University of Texas at Austin， where he graduated Phi Beta Kappa with a Bachelor's degree in Latin American Studies in 1973， taking only two and a half yearsto complete his work， and obtaining generally excellent grades.
4	生成句	was the University of California at Austin， where he studied in Beta Kappa in a degree of degree in English American Studies. 1968. and a one years a half yearsto complete. degree. which was a excellent grades.
5	目标句	Jeb Bushwas born inMidland， Texas， where his father was running an oil drilling company.
5	生成句	Bush， born intheway， Texas， and he fatherwas a a insurance company company.

References 35

[1]	Gibson M， Newman-Norlund R， Bonilha L， et al. The Aphasia Recovery Cohort， an open-source chronic stroke repository ［J］. Scientific Data， 2024， 11： No.981.
[2]	Peitz D， Schumann-Werner B， Hussmann K， et al. Success rates of intensive aphasia therapy： real-world data from 448 patients between 2003 and 2020 ［J］. Journal of Neurology， 2024， 271（11）： 7169-7183.
[3]	Le H， Lui F， Lui M Y. Aphasia［EB/OL］. （2024-10-29）［2025-09-02］..
[4]	Birbaumer N， Cohen L G. Brain-computer interfaces： communication and restoration of movement in paralysis ［J］. The Journal of Physiology， 2007， 579（3）： 621-636.
[5]	Metzger S L， Littlejohn K T， Silva A B， et al. A high-performance speech neuroprosthesis for speech decoding and avatar control ［J］. Nature， 2023， 620（7976）： 1031-1036.
[6]	Akbari H， Khalighinejad B， Herrero J L， et al. Towards reconstructing intelligible speech from the human auditory cortex［J］. Scientific Reports， 2019， 9： No.874.
[7]	Panachakel J T， Ramakrishnan A G. Decoding covert speech from EEG： a comprehensive review ［J］. Frontiers in Neuroscience， 2021， 15： No.642251.
[8]	Willett F R， Avansino D T， Hochberg L R， et al. High-performance brain-to-text communication via handwriting ［J］. Nature， 2021， 593（7858）： 249-254.
[9]	Moses D A， Metzger S L， Liu J R， et al. Neuroprosthesis for decoding speech in a paralyzed person with anarthria ［J］. The New England Journal of Medicine， 2021， 385（3）： 217-227.
[10]	陈景霞，郝为，张鹏伟，等. 基于混合神经网络的脑电时空特征情感分类［J］. 软件学报， 2021， 32（12）： 3869-3883.
	Chen Jingxia， Hao Wei， Zhang Pengwei， et al. Emotion classification of spatiotemporal EEG features using hybrid neural networks ［J］. Journal of Software， 2021， 32（12）： 3869-3883.
[11]	李锦瑶，杜肖兵，朱志亮，等. 脑电情绪识别的深度学习研究综述［J］. 软件学报， 2023， 34（1）： 255-276.
	Li Jingyao， Du Xiaobing， Zhu Zhiliang， et al. Deep learning for EEG-based emotion recognition： a survey ［J］. Journal of Software， 2023， 34（1）： 255-276.
[12]	Nicolas-Alonso L F， Gomez-Gil J. Brain-computer interfaces， a review ［J］. Sensors， 2012， 12（2）： 1211-1279.
[13]	Mahajan G， Divija L， Jeevan R， et al. A survey on deciphering of EEG waves ［C］// ISI 2023， SIST 389. Singapore： Springer， 2025： 391-404.
[14]	Srivastava A， Shinde T， Ansari R A， et al. Think2Type： thoughts to text using EEG waves ［J］. International Journal of Engineering Research， 2020， 9（6）： 659-664.
[15]	Wang Z， Ji H. Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification ［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2022， 36（5）： 5350-5358.
[16]	Lewis M， Liu Y， Goyal N， et al. BART： denoising sequence-to-sequence pre-training for natural language generation， translation， and comprehension ［C］// ACL 2020. Stroudsburg： ACL， 2020： 7871-7880.
[17]	Brigham K， Kumar B V K V. Imagined speech classification with EEG signals for silent communication： a preliminary investigation into synthetic telepathy ［C］// 2010 4th International Conference on Bioinformatics and Biomedical Engineering. Piscataway： IEEE， 2010： 1-4.
[18]	Wehbe L， Vaswani A， Knight K， Mitchell T. Aligning context-based statistical models of language with brain activity during reading ［C］// EMNLP 2014. Stroudsburg： ACL， 2014： 233-243.
[19]	代成龙，李光辉，李栋，等. 多项正则化约束的伪标签传播优化脑电信号聚类［J］. 计算机研究与发展， 2024， 61（1）： 156-171.
	Dai Chenglong， Li Guanghui， Li Dong， et al. Electroencephalogram clustering with multiple regularization constrained pseudo label propagation optimization ［J］. Journal of Computer Research and Development， 2024， 61（1）： 156-171.
[20]	Makin J G， Moses D A， Chang E F. Machine translation of cortical activity to text with an encoder-decoder framework ［J］. Nature Neuroscience， 2020， 23（4）： 575-582.
[21]	Hollenstein N， Rengli C， Glaus B， et al. Decoding EEG brain activity for multimodal natural language processing ［J］. Frontiers in Human Neuroscience， 2021， 15： No.659410.
[22]	Qiu J， Han W， Zhu J， et al. Can brain signals reveal inner alignment with human languages？［C］// Findings of the Association for Computational Linguistics： EMNLP 2023. Stroudsburg： ACL， 2023： 1789-1804.
[23]	Défossez A， Caucheteux C， Rapin J， et al. Decoding speech perception from non-invasive brain recordings ［J］. Nature Machine Intelligence， 2023， 5（10）： 1097-1107.
[24]	Amrani H， Micucci D， Napoletano P. Deep representation learning for open vocabulary electroencephalography-to-text decoding ［J］. IEEE Journal of Biomedical and Health Informatics， 2026， 30（4）： 3593-3600.
[25]	Duan Y， Zhou J， Wang Z， et al. DeWave： discrete encoding of EEG waves for EEG-to-text translation ［C］// NeurIPS 2023. Red Hook： Curran Associates Inc.， 2023： 9907-9918.
[26]	Schroff F， Kalenichenko D， Philbin J. FaceNet： a unified embedding for face recognition and clustering ［C］// CVPR 2015. Piscataway： IEEE， 2015： 815-823.
[27]	Sun B， Saenko K. Deep CORAL： correlation alignment for deep domain adaptation［C］// ECCV 2016. Cham： Springer， 2016： 443-450.
[28]	Hollenstein N， Rotsztejn J， Troendle M， et al. ZuCo， a simultaneous EEG and eye-tracking resource for natural sentence reading ［J］. Scientific Data， 2018， 5： No.180291.
[29]	Hollenstein N， Troendle M， Zhang C， et al. ZuCo 2.0： a dataset of physiological recordings during natural reading and annotation［C］// LREC 2020. Stroudsburg： ACL， 2020： 138-146.
[30]	Paszke A， Gross S， Massa F， et al. PyTorch： an imperative style， high-performance deep learning library ［C］// NeurIPS 2019. Red Hook： Curran Associates Inc.， 2019： 8026-8037.
[31]	Wolf T， Debut L， Sanh V， et al. Transformers： state-of-the-art natural language processing ［C］// EMNLP 2020： System Demonstrations. Stroudsburg： ACL， 2020： 38-45.
[32]	Raffel C， Shazeer N， Roberts A， et al. Exploring the limits of transfer learning with a unified text-to-text Transformer ［J］. Journal of Machine Learning Research， 2020， 21： 1-67.
[33]	Zhang J， Zhao Y， Saleh M， et al. PEGASUS： pre-training with extracted gap-sentences for abstractive summarization ［J］. Proceedings of Machine Learning Research， 2020， 119： 11328-11339.
[34]	Papineni K， Roukos S， Ward T， Zhu W J. BLEU： a method for automatic evaluation of machine translation ［C］// ACL 2002. Stroudsburg： ACL， 2002： 311-318.
[35]	Lin C Y. ROUGE： a package for automatic evaluation of summaries ［C］// ACL-04 Workshop： Text Summarization Branches Out. Stroudsburg： ACL， 2004： 74-81.