Optical anti-counterfeiting elements of diffractive optically variable image have been widely adopted in various high-security or high-value-added printed products such as banknotes, certificates/cards, and product packaging to prevent counterfeiting. This kind of optical anti-counterfeiting elements have optically variable characteristics, and three-dimensional and dynamically changing images generated by these elements under white light illumination conditions will cause serious interference to anomaly detection, further making it difficult to detect abnormal situations in industrial production process timely. In response to the challenge that the optical variable characteristics of diffractive optically variable images cannot be adapted by the existing unsupervised anomaly detection algorithms, a diffractive optically variable image anomaly detection algorithm based on Reverse Distillation (RD) was proposed. In the algorithm, noise was added to normal samples to generate pseudo-anomalous samples, and possible abnormal phenomena in industrial fields were simulated. Subsequently, both normal and pseudo-anomalous samples were input into the network as image pairs, and based on the Siamese network architecture, a Contrast and Reconstruction Module (CRM) was proposed. In this module, contrastive learning and reconstruction were conducted to the features extracted by the encoder from normal samples and pseudo-anomalous samples through feature reconstruction layer. This not only avoided the inflow of abnormal information into the decoder, leading to distillation failure, but also ensured that the reconstructed features conformed to the normal distribution of samples. Following this, the reconstructed features were input into feature fusion layer and feature compression layer for feature fusion and dimension reduction, and the compressed features were decoded layer by layer using the decoder. Finally, by using collaborative discrepancy optimization algorithm, the decoded features and the features extracted by the encoder were distilled to identify and locate abnormal information within the samples. Experimental results show that compared to the existing advanced anomaly detection algorithms on a certain anti-counterfeiting label dataset, the proposed algorithm improves adaptability to optically variable characteristics of diffractive optically variable images, and maintains high detection accuracy for abnormal regions within samples, achieving 100% in Image-Area Under Receiver Operator Curve (Image-AUROC), 95.02% in Pixel-Area Under Receiver Operator Curve (Pixel-AUROC), and 92.98% in Per-Region-Overlap (PRO). These results meet the requirements for anomaly detection of diffractive optically variable images in industrial fields.