基于提交排序和预测模型的测试套件选择方法

doi:10.11772/j.issn.1001-9081.2021061016

《计算机应用》唯一官方网站 ›› 2022, Vol. 42 ›› Issue (8): 2534-2539.DOI: 10.11772/j.issn.1001-9081.2021061016

• 计算机软件技术 • 上一篇

基于提交排序和预测模型的测试套件选择方法

刘美英, 杨秋辉(), 王潇, 蔡创

四川大学计算机学院，成都 610065

收稿日期:2021-06-16 修回日期:2021-09-02 接受日期:2021-11-17 发布日期:2022-01-25 出版日期:2022-08-10
通讯作者: 杨秋辉
作者简介:刘美英（1997—），女，四川广安人，硕士研究生，主要研究方向：软件质量保证与测试；
杨秋辉（1970—），女，山东青岛人，副教授，博士，CCF会员，主要研究方向：软件工程、软件项目管理；
王潇（1994—），女，湖北黄冈人，硕士，主要研究方向：软件质量保证与测试；
蔡创（1996—），男，四川大竹人，硕士研究生，主要研究方向：软件测试。

Test suite selection method based on commit prioritization and prediction model

Meiying LIU, Qiuhui YANG(), Xiao WANG, Chuang CAI

College of Computer Science，Sichuan University，Chengdu Sichuan 610065，China

Received:2021-06-16 Revised:2021-09-02 Accepted:2021-11-17 Online:2022-01-25 Published:2022-08-10
Contact: Qiuhui YANG
About author:LIU Meiying， born in 1997， M. S. candidate. Her research interests include software quality assurance and testing.
YANG Qiuhui， born in 1970， Ph. D.， associate professor. Her research interests include software engineering， software project management.
WANG Xiao， born in 1994， M. S. Her research interests include software quality assurance and testing.
CAI Chuang， born in 1996， M. S. candidate. His research interests include software testing.

摘要/Abstract

摘要：

为在持续集成（CI）环境下减少回归测试集、提升回归测试的效率，提出一种适用于CI环境的回归测试套件选择方法。首先，根据每个提交的测试套件历史失败率和执行率信息，进行提交排序；然后，采用机器学习方法，对提交涉及的测试套件进行失败率预测，并选择具有较高失败率的测试套件。该方法综合使用提交排序技术和测试套件选择技术，从而保证既提高故障检测率又能在一定程度上降低测试成本。在Google的开源数据集上进行的实验结果表明：与同样采用提交排序的方法和采用测试套件选择的方法相比，所提方法的开销感知平均故障检测率APFDc提高了1%~27%；在相同的测试时间成本下，所提方法的测试召回提高了33.33~38.16个百分点，变更召回提高了15.67~24.52个百分点，测试套件选择率降低了约6个百分点。

关键词: 持续集成, 回归测试, 提交排序, 测试套件选择, 测试套件失败率预测

Abstract:

In order to reduce the regression test set and improve the efficiency of regression test in the Continuous Integration （CI） environment， a regression test suite selection method for the CI environment was proposed. First， the commits were prioritized based on the historical failure rate and execution rate of each test suite related to each commit. Then， the machine learning method was used to predict the failure rates of the test suites involved in each commit， and the test suite with the higher failure rate were selected. In this method， the commit prioritization technology and the test suite selection technology were combined to ensure the increase of the failure detection rate and the reduction of the test cost. Experimental results on Google’s open-source dataset show that compared to the methods with the same commit prioritization method and test suite selection method， the proposed method has the highest improvement in the Average Percentage of Faults Detected per cost （APFDc） by 1% to 27%； At the same cost of test time， the TestRecall of this method increases by 33.33 to 38.16 percentage points， the ChangeRecall increases by 15.67 to 24.52 percentage points， and the test suite SelectionRate decreases by about 6 percentage points.

Key words: Continuous Integration (CI), regression test, commit prioritization, test suite selection, test suite failure rate prediction

中图分类号:

TP311.5

刘美英, 杨秋辉, 王潇, 蔡创. 基于提交排序和预测模型的测试套件选择方法[J]. 计算机应用, 2022, 42(8): 2534-2539.

Meiying LIU, Qiuhui YANG, Xiao WANG, Chuang CAI. Test suite selection method based on commit prioritization and prediction model[J]. Journal of Computer Applications, 2022, 42(8): 2534-2539.

图/表 7

图1 本文方法的流程

Fig. 1 Flowchart of the proposed method

图2 测试套件失败率预测和选择模型构建流程

Fig. 2 Construction flowchart of test suite failure rate prediction and selection model

表1 Google共享数据集

Tab. 1 Google shared datasets

测试阶段	提交数	测试套件执行记录	失败测试套件记录
总计	6 071	2 506 926	6 505
GooglePre	1 639	1 045 623	1 579
GooglePost	4 422	1 461 303	4 926

表2 不同时间执行率下不同方法的揭错率对比 (%)

Tab. 2 Comparison of error revealing rate under different methods

数据集	时间执行率	揭错率
数据集	时间执行率	文献［4］方法	文献［5］方法	本文方法
GooglePre	25	28	25	33
	50	53	55	69
	75	71	83	87
GooglePost	25	48	23	47
	50	68	48	69
	75	81	78	87

表3 不同方法的APFDc值对比

Tab. 3 APFDc value comparison

数据集	文献［4］方法	文献［5］方法	本文方法
GooglePre	0.470 3	0.497 8	0.596 5
GooglePost	0.561 6	0.552 7	0.567 2

图3 GooglePre数据集上TestChange、ChangeRecall和SelectionRate的对比

Fig. 3 Comparison of TestChange， ChangeRecall and SelectionRate on GooglePre dataset

图4 GooglePost数据集上TestChange、ChangeRecall和SelectionRate的对比

Fig. 4 Comparison chart of TestChange， ChangeRecall and SelectionRate on GooglePost dataset

参考文献 26

1	肖蕾.持续集成环境下基于历史执行信息的测试用例优先排序方法研究［D］.上海：上海大学， 2020： 100-120. 10.14419/ijet.v7i2.28.13207
	XIAO L. Research on test case prioritization technique based on history execution information in continuous integration ［D］. Shanghai： Shanghai University， 2020： 100-120. 10.14419/ijet.v7i2.28.13207
2	LEONG C， SINGH A， PAPADAKIS M， et al. Assessing transition-based test selection algorithms at Google ［C］// Proceedings of the 2019 ACM/IEEE 41st International Conference on Software Engineering： Software Engineering in Practice. New York： ACM， 2019： 101-110. 10.1109/icse-seip.2019.00019
3	王晓琳，曾红卫，林玮玮.敏捷开发环境中的回归测试优化技术［J］.计算机学报， 2019， 42（10）： 2323-2338. 10.11897/SP.J.1016.2019.02323
	WANG X L， ZENG H W， LIN W W. Techniques for regression testing in agile development environment［J］. Chinese Journal of Computers， 2019， 42（10）： 2323-2338. 10.11897/SP.J.1016.2019.02323
4	LIANG J J， ELBAUM S， ROTHERMEL G. Redefining prioritization： continuous prioritization for continuous integration ［C］// Proceedings of the 2018 IEEE/ACM 40th International Conference on Software Engineering. Piscataway： IEEE， 2018： 688-698. 10.1145/3180155.3180213
5	MACHALICA M， SAMYLKIN A， PORTH M， et al. Predictive test selection ［C］// Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering： Software Engineering in Practice： Software Engineering in Practice. Piscataway： IEEE， 2019： 91-100. 10.1109/icse-seip.2019.00018
6	MARIJAN D. Multi-perspective regression test prioritization for time-constrained environments ［C］// Proceedings of the 2015 IEEE International Conference on Software Quality， Reliability and Security. Piscataway： IEEE， 2015： 157-162. 10.1109/qrs.2015.31
7	MARIJAN D， LIAAEN M. Effect of time window on the performance of continuous regression testing ［C］// Proceedings of the 2016 IEEE International Conference on Software Maintenance and Evolution. Piscataway： IEEE， 2016： 568-571. 10.1109/icsme.2016.77
8	ENGSTRÖM E， RUNESON P， LJUNG A. Improving regression testing transparency and efficiency with history-based prioritization — an industrial case study ［C］// Proceedings of the 4th IEEE International Conference on Software Testing， Verification and Validation. Piscataway： IEEE， 2011： 367-376. 10.1109/icst.2011.27
9	CHO Y， KIM J， LEE E. History-based test case prioritization for failure information ［C］// Proceedings of the 23rd Asia-Pacific Software Engineering Conference. Piscataway： IEEE， 2016： 385-388. 10.1109/apsec.2016.066
10	SRIKANTH H， CASHMAN M， COHEN M B. Test case prioritization of build acceptance tests for an enterprise cloud application： an industrial case study［J］. Journal of Systems and Software， 2016， 119： 122-135. 10.1016/j.jss.2016.06.017
11	KIM J， JEONG H， LEE E. Failure history data-based test case prioritization for effective regression test ［C］// Proceedings of the 32nd Annual ACM Symposium on Applied Computing. New York： ACM， 2017： 1409-1415. 10.1145/3019612.3019831
12	李晋忠.基于日志的回归测试用例优先级排序系统的设计与实现［D］.北京：北京邮电大学， 2020： 70-90.
	LI J Z. Design and implementation of test case prioritization system for regression testing based on log ［D］. Beijing： Beijing University of Posts and Telecommunications， 2020： 70-90.
13	DO H， MIRARAB S， TAHVILDARI L， et al. The effects of time constraints on test case prioritization： a series of controlled experiments［J］. IEEE Transaction on Software Engineering， 2010， 36（5）： 593-617. 10.1109/tse.2010.58
14	EKELUND E D， ENGSTRÖM E. Efficient regression testing based on test history： an industrial evaluation ［C］// Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution. Piscataway： IEEE， 2015： 449-457. 10.1109/icsm.2015.7332496
15	MARIJAN D， LIAAEN M. Practical selective regression testing with effective redundancy in interleaved tests ［C］// Proceedings of the 2018 IEEE/ACM 40th International Conference on Software Engineering： Software Engineering in Practice Track. Piscataway： IEEE， 2018： 153-162. 10.1145/3183519.3183532
16	PHILIP A A， BHAGWAN R， KUMAR R， et al. FastLane： test minimization for rapidly deployed large-scale online services ［C］// Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering. Piscataway： IEEE， 2019： 408-418. 10.1109/icse.2019.00054
17	刘刚，黎放，狄鹏.基于融合算法的测试优化选择问题研究［J］.计算机科学， 2013， 40（S1）： 54-57. 10.3969/j.issn.1002-137X.2013.z1.012
	LIU G， LI F， DI P. Research on optimal test selection based on fused algorithm［J］. Computer Science， 2013， 40（S1）： 54-57. 10.3969/j.issn.1002-137X.2013.z1.012
18	王曙燕，陈朋媛，孙家泽.基于变异分析的测试用例约简方法［J］.计算机应用， 2017， 37（12）： 3592-3596. 10.11772/j.issn.1001-9081.2017.12.3592
	WANG S Y， CHEN P Y， SUN J Z. Reduction method of test suites based on mutation analysis［J］. Journal of Computer Applications， 2017， 37（12）： 3592-3596. 10.11772/j.issn.1001-9081.2017.12.3592
19	ELBAUM S， ROTHERMEL G， PENIX J. Techniques for improving regression testing in continuous integration development environments ［C］// Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. New York： ACM， 2014： 235-245. 10.1145/2635868.2635910
20	SPIEKER H， GOTLIEB A， MARIJAN D， et al. Reinforcement learning for automatic test case prioritization and selection in continuous integration ［C］// Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. New York： ACM， 2017： 12-22. 10.1145/3092703.3092709
21	NAJAFI A， SHANG W Y， RIGBY P C. Improving test effectiveness using test executions history： an industrial experience report ［C］// Proceedings of the 2019 IEEE/ACM 41th International Conference on Software Engineering： Software Engineering in Practice. Piscataway： IEEE， 2019： 213-222. 10.1109/icse-seip.2019.00031
22	ALI S I， SHAHZAD W. A feature subset selection method based on symmetric uncertainty and Ant Colony Optimization ［C］// Proceedings of the 2012 International Conference on Emerging Technologies. Piscataway： IEEE， 2012： 1-6. 10.1109/icet.2012.6375420
23	CHEN T Q， GUESTRIN C. XGBoost： a scalable tree boosting system ［C］// Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2016： 785-794. 10.1145/2939672.2939785
24	ELBAUM S， MCLAUGHLIN A， PENIX J. The Google dataset of testing results［EB/OL］. （2013-12-19）［2021-03-10］. .
25	ELBAUM S， MALISHEVSKY A， ROTHERMEL G. Incorporating varying test costs and fault severities into test case prioritization ［C］// Proceedings of the 23rd International Conference on Software Engineering. Piscataway： IEEE， 2001： 329-338. 10.1109/32.988497
26	MEMON A， GAO Z B， NGUYEN B， et al. Taming Google-scale continuous testing ［C］// Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering： Software Engineering in Practice. Piscataway： IEEE， 2017： 233-242. 10.1109/icse-seip.2017.16

[1]	王曙燕, 袁佳娟, 孙家泽. 基于弱变异准则的测试用例约简方法[J]. 计算机应用, 2019, 39(2): 534-539.
[2]	王曙燕, 陈朋媛, 孙家泽. 基于变异分析的测试用例约简方法[J]. 计算机应用, 2017, 37(12): 3592-3596.
[3]	王曙燕, 温春琰, 孙家泽. 基于自适应粒子群优化算法的测试数据扩增方法[J]. 计算机应用, 2016, 36(9): 2492-2496.
[4]	邢行, 尚颖, 赵瑞莲, 李征. 面向多目标测试用例优先排序的蚁群算法信息素更新策略[J]. 计算机应用, 2016, 36(9): 2497-2502.
[5]	杜章华李建华胡江明程晓菊. 基于方法切片及入口依赖的回归测试研究[J]. 计算机应用, 2010, 30(06): 1651-1654.
[6]	陈树峰郑洪源. 面向对象软件的依赖性分析与回归测试[J]. 计算机应用, 2009, 29(11): 3110-3113.
[7]	陆永忠宋骏礼古希谦. 基于行为的软件测试过程模型及其应用研究[J]. 计算机应用, 2007, 27(5): 1238-1240.
[8]	尤永康，刘乃琦. 自动化回归测试在Java项目中的实现[J]. 计算机应用, 2005, 25(01): 88-90.

基于提交排序和预测模型的测试套件选择方法

Test suite selection method based on commit prioritization and prediction model

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 26

相关文章 8

编辑推荐

Metrics