Abstract: The traditional face annotation methods for news images mainly rely on similarity information of the faces, and have poor ability to distinguish non-noise faces from noise faces and to annotate non-noise faces. Aiming at this issue, a face annotation method based on multi-modal information fusion was proposed. Firstly, according to the co-occurrence relations between faces and names, face-name match degrees based on face similarity were obtained by using a modified K-Nearest Neighbor (KNN) algorithm. After that, face importance degrees were characterized by the size and position information of faces extracted from images, and name importance degrees were characterized by the name position information extracted from images. Finally, Back Propagation (BP) neural network was applied to fuse the above information to infer labels of faces, and an annotation result correcting strategy was proposed to further improve the annotation results. Experimental results on Label Yahoo!News dataset demonstrate that the accuracy, precision and recall of the proposed method reach 77.11%, 73.58% and 78.75% respectively; compared with the methods only based on face similarity, the proposed method has outstanding ability to distinguish non-noise faces from noise faces and to annotate non-noise faces.

Key words: news image, faces annotation, K-Nearest Neighbor (KNN) algorithm, multi-modal information, Back Propagation (BP) neural network

摘要： 针对传统新闻图像中人脸标注方法主要依赖人脸相似度信息，分辨噪声和非噪声人脸能力以及非噪声人脸标注能力较差的问题，提出一种基于多模态信息融合的新闻图像人脸标注方法。首先根据人脸和姓名的共现关系，利用改进的K近邻算法，获得基于人脸相似度信息的人脸姓名匹配度；然后，分别从图像中提取人脸大小和位置的信息对人脸重要程度进行表征，从文本中提取姓名位置信息对姓名重要程度进行表征；最后，使用反向传播神经网络来融合上述信息完成人脸标签的推理，并提出一个标签修正策略来进一步改善标注结果。在Label Yahoo! News数据集上的测试效果表明，所提方法的标注准确率、精度和召回率分别达到了77.11%、73.58%和78.75%，与仅基于人脸相似度的算法相比，具有较好的分辨噪声和非噪声人脸能力以及非噪声人脸标注能力。

关键词: 新闻图像, 人脸标注, K近邻算法, 多模态信息, 反向传播神经网络