Abstract: Decoding text from electroencephalography (EEG) is a hotspot in brain-computer interface (BCI) research. Existing methods focus on global context modeling, neglect inter-channel local correlations, fail to realize their synergy, and leave EEG-text representation alignment in public semantic mapping space unsolved. To address these limitations, this study proposes a dual-branch EEG decoding framework with a cross-modal alignment strategy. The spatiotemporal branch sequentially employs a bidirectional long short-term memory network, depthwise separable convolution, and gated axial self-attention to extract short-term dependencies, spatial relationships across adjacent channels, and long-range temporal interactions. In parallel, the context branch uses a multi-layer Transformer encoder to model global context. A cross-attention module integrates the features from both branches. For cross-modal alignment, the framework introduces a joint loss combining triplet loss and covariance alignment. The triplet loss constrains the geometric distance between EEG and text pairs, while the covariance alignment ensures consistency in second-order statistical properties of the two modalities. This design effectively narrows the semantic gap between EEG and text representations. Experiments on the ZuCo dataset show that the model improves BLEU-1 by approximately 1% over a strong baseline, validating its effectiveness in EEG-to-text decoding.

Key words: Electroencephalography, deep neural network, feature fusion, modality discrepancy, text generation

摘要： 摘 要: 将脑电信号（EEG）解码可读文本是脑-机接口（BCI）领域的研究热点，现有方法偏重于全局上下文建模、忽视信号通道间局部关联且未能实现二者协同；同时EEG与文本表征在公共语义映射空间的对齐问题未有效解决。为此，该文提出一种基于双分支表征融合与跨模态对齐的EEG解码模型。信号编码阶段采用并行双分支架构：时空结构建模分支通过双向长短期记忆网络、深度可分离卷积与门控轴向自注意力机制，分层捕获信号的短时依赖、邻近通道空间相关性与跨步长程关系；上下文融合分支则基于多层Transformer编码器，借交叉注意力融合两路表征以互补整合。跨模态对齐机制引入三元组损失与协方差对齐联合损失，分别从EEG与文本表征样本对间几何距离及二阶统计特性约束向量对齐，弱化模态语义鸿沟。公开ZuCo数据集实验表明，该模型在BLEU-1指标上较主流基线提高约1个百分点，验证了其在面向文本的EEG解码任务中的有效性。

关键词: 电图, 深度神经网络, 特征融合, 模态差异性, 文本生成