计算机应用

• 人工智能与仿真 •    下一篇

基于图像识别的移动端应用控件检测方

张文烨   

  1. 中国人民大学 信息学院
  • 收稿日期:2019-08-15 修回日期:2019-10-30 发布日期:2019-10-30 出版日期:2020-05-12
  • 通讯作者: 张文烨
  • 作者简介:张文烨(1994—),女,硕士研究生,山西太原人,主要研究方向:软件自动化测试、计算机视觉。

APP component recognition method based on object detection

ZHANG Wenye   

  • Received:2019-08-15 Revised:2019-10-30 Online:2019-10-30 Published:2020-05-12
  • Contact: Wenye .Zhang

摘要: 为解决在浮窗等页面和控件重叠场景下,目前移动端应用常用的自动化测试技术无法准确识别待测控件的问题,基于计算机视觉和目标检测技术,提出一种新的移动端应用控件识别方法,将控件识别任务变换为目标检测任务。其次,采集应用的交互界面并标注其中的待测控件位置及类型,并使用多种主流目标检测模型在该数据集上进行训练和验证。最后,实现目标检测结果至 Appium测试脚本的自动转换,使得基于目标检测技术的测试工具完全自动化运行成为可能。实验表明,所提方法可以有效识别待测控件,体现了对应用运行平台(Android/iOS)的鲁棒性,最高可取得 0. 634的平均精度均值(mAP)。

关键词: GUI测试, 自动化测试, 控件识别, 目标检测

Abstract: In the overlapping scenes of pages and components such as popup-windows,the current automated testing technology commonly used in mobile applications cannot accurately identify the components to be tested. To solve thses probelms,a new type of automated testing technology was proposed based on object detection method,and the component detection task would be transformed to object detection task. Secondly,a variety of mainstream object detection models were trained and validated on the dataset,which consists of the components with types and locations marked in GUI. Finally,the automatic conversion of the Appium test script from the detection result was implemented,which makes it possible to fully automate the test tool based on the object detection method. Experiment results show that the proposed method could effectively identify the components to be tested,also obtained a mAP(mean Average Precision)score of 0. 634. At the same time,the detection result shown the robustness to the operating system(Android/iOS).

Key words: GUI testing, automated testing, component identification, object detection

中图分类号: