Survey of research progress on crowdsourcing task assignment for evaluation of workers’ ability

doi:10.11772/j.issn.1001-9081.2020101629

Abstract

Abstract: With the rapid development of internet technology and sharing economy mode, as a new crowd computing mode, crowdsourcing has been widely applied and become a research focus recently. Aiming at the characteristics of crowdsourcing applications, to ensure the completion quality of crowdsourcing tasks, the existing researches have proposed different crowdsourcing task assignment methods from the perspective of the evaluation of worker's ability. Firstly, the crowdsourcing's concept and classification were introduced, and the workflow and task characteristics of the crowdsourcing platform were analyzed. Based on them, the existing research works on the evaluation of workers' ability were summarized. Then, the crowdsourcing task assignment methods and the related challenges were reviewed from three different aspects, including matching-based, planning-based and role-based collaboration. Finally, the research directions of future work were put forward.

Key words: crowdsourcing, task assignment, evaluation of workers’ ability, quality control, information security

摘要： 随着互联网技术和共享经济模式的快速发展，作为一种新型的群体计算模式，近年来众包（Crowdsourcing）已经得到了广泛的应用并成为研究热点。针对众包应用的特点，为确保众包任务的完成质量，现有研究从工作者能力评估的角度出发已提出了各种不同的众包任务分配方法。首先介绍了众包的概念和分类，阐述了众包平台的工作流程及其任务特点，并在此基础上总结了现有的工作者能力评估的相关研究工作；然后从基于匹配、基于规划和基于角色协同等三个方面综述了众包任务分配方法及其遇到的挑战；最后提出了未来工作的研究方向。

关键词: 众包, 任务分配, 工作者能力评估, 质量控制, 信息安全

CLC Number:

TP311

MA Hua, CHEN Yuepeng, TANG Wensheng, LOU Xiaoping, HUANG Zhuoxuan. Survey of research progress on crowdsourcing task assignment for evaluation of workers’ ability[J]. Journal of Computer Applications, 2021, 41(8): 2232-2241.

马华, 陈跃鹏, 唐文胜, 娄小平, 黄卓轩. 面向工作者能力评估的众包任务分配方法的研究进展综述[J]. 《计算机应用》唯一官方网站, 2021, 41(8): 2232-2241.

References

[1] 王鸿飞, 陈历敏, 何静. 科研众包平台发展现状与对策——基于国际、国内、广东省科研众包培育平台案例的分析[J]. 科技创新发展战略研究,2019,3(5):6-19.(WANG H F,CHEN L M,HE J. Development status and countermeasures of scientific research crowdsourcing platform:based on the case study of international, domestic and Guangdong crowdsourcing platforms[J]. Strategy for Innovation and Development of Science and Technology,2019,3(5):6-19.)
[2] ALLAHBAKHSH M,BENATALLAH B,IGNJATOVIC A,et al. Quality control in crowdsourcing systems:issues and directions[J]. IEEE Internet Computing,2013,17(2):76-81.
[3] 中国计算机学会. CCF 2016-2017中国计算机科学技术发展报告[M]. 北京:机械工业出版社,2017:10.(China Computer Federation. CCF 2016-2017 China Computer Science and Technology Development Report[M]. Beijing:China Machine Press,2017:10.)
[4] MA S R,WANG L,GUO J E,et al. Research on task pricing of crowdsourcing platform[C]//Proceedings of the 2018 International Conference on Computer Science,Electronics and Communication Engineering. Paris:Atlantis Press,2018:518-521.
[5] HUNG N Q V,THANG D C,TAM N T,et al. Answer validation for generic crowdsourcing tasks with minimal efforts[J]. The VLDB Journal,2017,26(6):855-880.
[6] LI G L, ZHENG Y D, FAN J, et al. Crowdsourced data management:Overview and challenges[C]//Proceedings of the 2017 ACM International Conference on Management of Data. New York:ACM,2017:1711-1716.
[7] MAVRIDIS P, GROSS-AMBLARD D, MIKLÓS Z. Using hierarchical skills for optimized task assignment in knowledgeintensive crowdsourcing[C]//Proceedings of the 25th International Conference on World Wide Web. Republic and Canton of Geneva:International World Wide Web Conferences Steering Committee, 2016:843-853.
[8] CHEN L, LEE D, MILO T. Data-driven crowdsourcing:management,mining,and applications[C]//Proceedings of the IEEE 31st International Conference on Data Engineering. Piscataway:IEEE,2015:1527-1529.
[9] DOAN A,FRANKLIN M J,KOSSMANN D,et al. Crowdsourcing applications and platforms:A data management perspective[J]. Proceeding of the VLDB Endowment,2011,4(12):1508-1509.
[10] 童咏昕, 袁野, 成雨蓉, 等. 时空众包数据管理技术研究综述[J]. 软件学报,2017,28(1):35-58.(TONG Y X,YUAN Y, CHENG Y R,et al. Survey on spatiotemporal crowdsourced data management techniques[J]. Journal of Software,2017,28(1):35-38.)
[11] 冯剑红, 李国良, 冯建华. 众包技术研究综述[J]. 计算机学报, 2015,38(9):1713-1726.(FENG J H,LI G L,FENG J H. A survey on crowdsourcing[J]. Chinese Journal of Computers, 2015,38(9):1713-1726.)
[12] 潘庆先, 江珊, 董红斌, 等. 基于Pareto分布的众包工人欺骗行为处理方法[J]. 计算机应用,2019,39(11):3191-3197. (PAN Q X,JIANG S,DONG H B,et al. Pareto distribution based processing approach of deceptive behaviors of crowdsourcing workers[J]. Journal of Computer Applications,2019,39(11):3191-3197.)
[13] 张志强, 逄居升, 谢晓芹, 等. 众包质量控制策略及评估算法研究[J]. 计算机学报,2013,36(8):1636-1649.(ZHANG Z Q, PANG J S,XIE X Q,et al. Research on crowdsourcing quality control strategies and evaluation algorithm[J]. Chinese Journal of Computers,2013,36(8):1636-1649.)
[14] 陈跃鹏. 基于工人能力模糊度量的软件众包任务协同分配研究[D]. 长沙:湖南师范大学,2020:1-20.(CHEN Y P. Research on collaborative assignment of software crowdsourcing task based on fuzzy measurement of workers'capability[D]. Changsha:Hunan Normal University,2020:1-20.)
[15] HOWE J. The rise of crowdsourcing[J]. Wired Magazine,2006, 14(6):176-183.
[16] ESTELLÉS-AROLAS E, GONZÁLEZ-LADRÓN-DE-GUEVARA F. Towards an integrated crowdsourcing definition[J]. Journal of Information Science,2012,38(2):189-200.
[17] KIETZMANN J H. Crowdsourcing:a revised definition and introduction to new research[J]. Business Horizons,2017,60(2):151-153.
[18] ZHAO Y X,ZHU Q H. Evaluation on crowdsourcing research:current status and future direction[J]. Information Systems Frontiers,2014,16(3):417-434.
[19] SAXTON G D,OH O,KISHORE R. Rules of crowdsourcing:models,issues,and systems of control[J]. Information Systems Management,2013,30(1):2-20.
[20] BRABHAM D C. Crowdsourcing as a model for problem solving:an introduction and cases[J]. Convergence,2008,14(1):75-90.
[21] 尹刚, 王涛, 刘冰珣, 等. 面向开源生态的软件数据挖掘技术研究综述[J]. 软件学报,2018,29(8):2258-2271.(YIN G, WANG T,LIU B X,et al. Survey of software data mining for open source ecosystem[J]. Journal of Software,2018,29(8):2258-2271.)
[22] 谢新强, 杨晓春, 王斌, 等. 一种多特征融合的软件开发者推荐方法[J]. 软件学报,2018,29(8):2306-2321.(XIE X Q, YANG X C, WANG B, et al. Multi-feature fused software developer recommendation[J]. Journal of Software,2018,29(8):2306-2321.)
[23] JURCA R, FALTINGS B. Towards incentive-compatible reputation management[C]//Proceedings of the 2002 Workshop on Deception,Fraud and Trust in Agent Societies,LNCS 2631. Berlin:Springer,2002:138-147.
[24] 严俊, 库少平, 喻楚. 基于活跃度的众包工作者信誉模型[J]. 计算机应用,2017,37(7):2039-2043.(YAN J,KU S P,YU C. Reputation model of crowdsourcing workers based on active degree[J]. Journal of Computer Applications,2017,37(7):2039-2043.)
[25] LU J C,LIAO J. Notice of retraction:e-commerce customer credit evaluation based on AHP[C]//Proceedings of the 2010 IEEE International Conference on Software Engineering and Service Sciences. Piscataway:IEEE,2010:655-658.
[26] SHAO K,GUO Y J,LIU C H. The model of credit evaluation in C2C e-commerce[C]//Proceedings of the 2nd IEEE International Conference on Information Management and Engineering. Piscataway:IEEE,2010:118-121.
[27] 余敦辉, 王意, 张万山. 软件众包工人能力动态度量算法[J]. 计算机应用,2018,38(12):3612-3617.(YU D H,WANG Y, ZHANG W S. Ability dynamic measurement algorithm for software crowdsourcing workers[J]. Journal of Computer Applications, 2018,38(12):3612-3617.)
[28] CROWSTON K,WEI K N,LI Q,et al. Core and periphery in free/libre and open source software team communications[C]//Proceedings of the 39th Annual Hawaii International Conference on System Sciences. Piscataway:IEEE,2006:No. 118a.
[29] MOCKUS A,FIELDING R T,HERBSLEB J D. Two case studies of open source software development:Apache and Mozilla[J]. ACM Transactions on Software Engineering and Methodology, 2002,11(3):309-346.
[30] 吴哲夫, 朱天潼, 宣琦, 等. 基于贡献分配的开源软件核心开发者评估研究[J]. 软件学报,2018,29(8):2272-2282.(WU Z F,ZHU T T,XUAN Q,et al. Evaluation of core developers in open source software by contribution allocation[J]. Journal of Software,2018,29(8):2272-2282.)
[31] EICKHOFF C,DE VRIES A P. Increasing cheat robustness of crowdsourcing tasks[J]. Information Retrieval,2013,16(2):121-137.
[32] EICKHOFF C,DE VRIES A P. How crowdsourcable is your task?[C]//Proceedings of the 2011 Workshop on Crowdsourcing for Search and Data Mining at the 4th ACM International Conference on Web Search and Data Mining. New York:ACM,2011:11-14.
[33] KITTUR A,CHI E H,SUH B. Crowdsourcing user studies with Mechanical Turk[C]//Proceedings of the 2008 SIGCHI Conference on Human Factor in Computing Systems. New York:ACM,2008:453-456.
[34] BERNSTEIN M S,BRANDT J,MILLER R C,et al. Crowds in two seconds:enabling realtime crowd-powered interfaces[C]//Proceedings of the 24th ACM Symposium on User Interface Software and Technology. New York:ACM,2011:33-42.
[35] OLESON D,SOROKIN A,LAUGHLIN G,et al. Programmatic gold:targeted and scalable quality assurance in crowdsourcing[C]//Proceedings of the 2011 AAAI Workshops at the 25th AAAI Conference on Artificial Intelligence. Palo Alto, CA:AAAI Press,2011:43-48.
[36] GAO J Y,LIU X,OOI B C,et al. An online cost sensitive decision-making method in crowdsourcing systems[C]//Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. New York:ACM,2013:217-228.
[37] IPEIROTIS P G,PROVOST F,WANG J. Quality management on Amazon Mechanical Turk[C]//Proceedings of the 2010 ACM SIGKDD Workshop on Human Computation. New York:ACM, 2010:64-67.
[38] DAWID A P,SKENE A M. Maximum likelihood estimation of observer error-rates using the EM algorithm[J]. Journal of the Royal Statistical Society:Series C(Applied Statistics),1979,28(1):20-28.
[39] RAYKAR V C,YU S. An entropic score to rank annotators for crowdsourced labeling tasks[C]//Proceedings of the 3rd National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics. Piscataway:IEEE,2011:29-32.
[40] CAO C C,SHE J Y,TONG Y X,et al. Whom to ask?:jury selection for decision making tasks on micro-blog services[J] Proceedings of the VLDB Endowment,2012,5(11):1495-1506.
[41] HIRTH M,HOßFELD T,TRAN-GIA P. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms[J]. Mathematical and Computer Modelling,2013,57(11/12):2918-2932.
[42] QUINN A J,BEDERSON B B. A survey and taxonomy of a growing field[C]//Proceedings of the 2011 SIGCHI Conference on Human Factors in Computing Systems. New York:ACM,2011:1403-1412.
[43] 施战, 辛煜, 孙玉娥, 等. 基于用户可靠性的众包系统任务分配机制[J]. 计算机应用,2017,37(9):2449-2453.(SHI Z,XIN Y,SUN Y E,et al. Task allocation mechanism for crowdsourcing system based on reliability of users[J]. Journal of Computer Applications,2017,37(9):2449-2453.)
[44] CORMEN T H, LEISERSON C E, RIVEST R L, et al. Introduction to Algorithms[M]. 3rd ed. Cambridge:MIT Press, 2009:664-668.
[45] 徐哲, 李卓, 陈昕. 面向移动群智感知的多任务分发算法[J].计算机应用,2017,57(1):18-23,47.(XU Z,LI Z,CHEN X. Multi-task assignment algorithm for mobile crowdsensing[J]. Journal of Computer Applications,2017,37(1):18-23,47.)
[46] 张晓航, 李国良, 冯建华. 大数据群体计算中用户主题感知的任务分配[J]. 计算机研究与发展,2015,52(2):309-317. (ZHANG X H,LI G L,FENG J H. Theme-aware task assignment in crowd computing on big data[J]. Journal of Computer Research and Development,2015,52(2):309-317.)
[47] 王青, 谭良. 基于用户主题精确感知大数据群体计算任务分配算法[J]. 计算机应用,2016,36(10):2777-2783.(WANG Q, TAN L. Optimization algorithm for accurately theme-aware task assignment in crowd computing on big data[J]. Journal of Computer Applications,2016,36(10):2777-2783.)
[48] KAZEMI L,SHAHABI C. GeoCrowd:enabling query answering with spatial crowdsourcing[C]//Proceedings of the 20th International Conference on Advances in Geographic Information Systems. New York:ACM,2012:189-198.
[49] TONG Y X,SHE J Y,DING B L,et al. Online mobile micro-task allocation in spatial crowdsourcing[C]//Proceedings of the IEEE 32nd International Conference on Data Engineering. Piscataway:IEEE,2016:49-60.
[50] BOUTSIS I,KALOGERAKI V. Crowdsourcing under real-time constraints[C]//Proceedings of the IEEE 27th International Symposium on Parallel and Distributed Processing. Piscataway:IEEE,2013:753-764.
[51] TING H F,XIANG X Z. Near optimal algorithms for online maximum edge-weighted b-matching and two-sided vertex-weighted b-matching[J]. Theoretical Computer Science,2015,607(Pt 2):247-256.
[52] 余敦辉, 张灵莉, 付聪. 基于动态效用的时空众包在线任务分配[J]. 电子与信息学报,2018,40(7):1699-1706.(YU D H, ZHANG L L, FU C. Online task allocation of spatial crowdsourcing based on dynamic utility[J]. Journal of Electronics and Information Technology,2018,40(7):1699-1706.)
[53] TONG Y X,SHE J Y,DING B L,et al. Online minimum matching in real-time spatial data:experiments and analysis[J]. Proceedings of the VLDB Endowment,2016,9(12):1053-1064.
[54] 宋天舒, 童咏昕, 王立斌, 等. 空间众包环境下的3类对象在线任务分配[J]. 软件学报,2017,28(3):611-630.(SONG T S, TONG Y X,WANG L B,et al. Online task assinment for three type of objects under spatial crowdsourcing environment[J]. Journal of Software,2017,28(3):611-630.)
[55] HO C J, VAUGHAN J W. Online task assignment in crowdsourcing markets[C]//Proceedings of the 26th AAAI Conference on Artifical Intelligence. Palo Alto,CA:AAAI Press, 2012:45-51.
[56] 蒋丽, 王静, 梁昌勇, 等. 基于改进蚁群算法的众包配送路径研究[J]. 计算机工程与应用,2019,55(8):244-249.(JIANG L, WANG J, LIANG C Y, et al. Research on crowdsourcing distribution path based on improved ant colony algorithm[J]. Computer Engineering and Applications, 2019, 55(8):244-249.)
[57] LI Y,YIU M L,XU W J. Oriented online route recommendation for spatial crowdsourcing task workers[C]//Proceedings of the 2015 International Symposium on Spatial and Temporal Databases,LNCS 9239. Cham:Springer,2015:137-156.
[58] 李洋, 贾梦迪, 杨文彦, 等. 基于树分解的空间众包最优任务分配算法[J]. 软件学报,2018,29(3):824-838.(LI Y,JIA M D,YANG W Y,et al. Optimal task assignment algorithm based on tree-decouple in spatial crowdsourcing[J]. Journal of Software, 2018,29(3):824-838.)
[59] DENG D X,SHAHABI C,DEMIRYUREK U. Maximizing the number of worker's self-selected tasks in spatial crowdsourcing[C]//Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. New York:ACM,2013:324-333.
[60] DENG D X, SHAHABI C, ZHU L H. Task matching and scheduling for multiple workers in spatial crowdsourcing[C]//Proceedings of the 23rd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. New York:ACM,2015:No. 21.
[61] ZHU H B,ZHOU M C,ALKINS R. Group role assignment via a Kuhn-Munkres algorithm-based solution[J]. IEEE Transactions on Systems, Man, and Cybernetics-Part A:Systems and Humans,2012,42(3):739-750.
[62] ZHU H B,SHENG Y,ZHOU X Z,et al. Group role assignment with cooperation and conflict factors[J]. IEEE Transactions on Systems, Man, and Cybernetics:Systems, 2018, 48(6):851-863.
[63] MA H,CHEN Y P,ZHU H B,et al. Optimization of cloud service composition for data-intensive applications via E-CARGO[C]//Proceedings of the IEEE 22nd International Conference on Computer Supported Cooperative Work in Design. Piscataway:IEEE,2018:785-789.
[64] 周侨, 方明. 基于多Agent的众包任务分配算法的研究[J]. 智能计算机与应用,2019,9(1):104-107.(ZHOU Q,FANG M. Research on crowdsourcing task allocation algorithm based on multi-Agent[J]. Intelligent Computer and Applications,2019,9(1):104-107.)