Automated Modulation Recognition （AMR） plays a critical role in wireless communications. A Denoising & Dual-modal Attention CNN-Transformer （D-DmACT） was proposed to address problems of poor transfer ability and insufficient abilities to distinguish noise and modulation signal features of AMR networks in complex electromagnetic environments. Firstly， a generator to generate complex electromagnetic interference iteratively and a discriminator to be against interference were proposed to enhance generalization ability of the network when encountering complex electromagnetic environments. Secondly， a complex attention-based Transformer module was designed to capture time-domain features of In-phase and Quadrature （IQ） signals， and a coordination attention module based on time-frequency information was proposed to acquire features of time-frequency images， then the features were crossed and fused. Thirdly， temporal sequence complex signals and time-frequency image obtained by the generator were sent to the dual-modal attention fusion model. Finally， lightweight classification and recognition were implemented. Experiments were conducted on datasets RadioML2016.10A and RadioML2018.01a under white Gaussian noise and complex electromagnetic environment， respectively. Experimental results with impulsive noise show that compared with CLDNN（Convolutional Long short-term Deep Neural Networks）， Residual Network （ResNet）， and LSTM（Long Short-Term Memory） Network， the proposed network has the average recognition accuracy increased by 53.98%， 28.82%， and 24.64%； compared with Multi-Modal approach toward AMR （MM-Net）， Threshold Autoencoder Denoiser Convolutional Neural Network （TADCNN）， and Generative Adversarial Network & Multi-modal Attention mechanism CNN-LSTM （GAN-MnACL）， the proposed network has the average recognition accuracy enhanced by 19.74%， 13.55% and 11.17%， respectively. In terms of computational complexity， deployability of the proposed network is validated through metrics such as parameters and FLoating point OPerations （FLOPs）.