Abstract: Traditional methods for predicting efficacy in patients with locally advanced rectal cancer (LARC) often rely on single-modality medical imaging information, clinical data, or pathological information. To address this limitation, a novel multimodal medical image fusion network framework based on attention mechanisms was proposed to predict efficacy. This framework comprises a feature extraction module, a multi-scale feature fusion module, a multi-modal medical image fusion module, and an attention-based decision module. Initially, multi-scale deep features are extracted using the feature extraction and multi-scale feature fusion modules. These deep features are then integrated through the multi-modal medical image fusion module. Finally, the attention-based decision module combines clinical features, radiomics features, and deep features to predict treatment efficacy. Experimental results demonstrate that the proposed method improves accuracy by 5 percentage points, recall by 6 percentage points, and AUC by 4 percentage points on the experimental dataset compared to single-modality methods, thereby validating the significant enhancement in rectal cancer treatment efficacy prediction achieved by this method.

Key words: rectal cancer, efficacy prediction, attention mechanism, deep learning, medical image fusion

摘要： 针对传统方法预测局部晚期直肠癌(LARC)患者治疗疗效通常仅依赖于单一模态医学图像信息、临床信息或病理信息等问题，提出一种基于注意力机制的多模态医学图像融合网络框架，用于预测直肠癌疗效。该网络框架包含特征提取模块、多尺度特征融合模块、多模态医学图像融合模块和注意力决策模块。首先使用特征提取模块、多尺度特征融合模块提取多尺度深度特征，随后使用多模态医学图像融合模块融合深度特征，最后通过注意力决策模块将临床特征、影像组学特征和深度特征相结合进行疗效预测。实验结果表明，与单模态方法相比，所提方法在实验数据集上的准确率提升了5个百分点，召回率提升了6个百分点，AUC提升了4个百分点，从而验证了本方法能显著改善直肠癌疗效预测的结果。

关键词: 直肠癌, 疗效预测, 注意力机制, 深度学习, 医学图像融合