Abstract: In the task of handwriting identification, the large area of image is background, handwriting information is sparse, key information is difficult to capture, and personal handwriting signature style has slight changes and handwriting imitated is highly similar, as well as the lack of public Chinese handwriting datasets. By improving attention mechanism and Siamese network model, a handwriting identification Network based on the Multi-scale and Mixed Domain Attention mechanism (MMDANet) was proposed. Firstly, a maximum pooling layer was connected in parallel to the effective channel attention module, and the number of channels of two-dimensional strip pooling module was extended to three dimensions, and the two were fused to generate a Mixed Domain Module (MDM), thereby solving the problem that large area of handwriting image is background, handwriting information is sparse and detailed features are difficult to extract. Secondly, the Path Aggregation Network (PANet) feature pyramid was used to extract features at multiple scales to capture the subtle differences between true and false handwriting, and the comparison loss of Siamese network and the weighted fusion of Additive Margin Softmax (AM-Softmax) loss were used for training. It increased the degree of discrimination between categories, and solved the problem of personal handwriting style variation and high similarity between true and false handwriting. Finally, in order to promote the development of Chinese handwriting datasets, a Chinese Handwriting Dataset (CHD) with a total sample of 8,000 pieces was self-made. The accuracy rate of the proposed method on the Chinese dataset CHD reaches 84.25%, and compared with the suboptimal method Two-stage Siamese Network (Two-stage SiamNet), the proposed method is accurate on three foreign language datasets Cedar, Bengla and Hindi. The rates increased by 4.53%, 1.02% and 1.67% respectively.The experimental results show that the MMDANet can more accurately capture the subtle differences between true and false handwriting, and complete complex handwriting identification tasks.