关键词: Spark, 实时数据流, 多应用, 动态资源分配

Abstract: With the development of science and technology, large data applications on SaaS model platform are complex and diverse, may also contain different characteristics of the data and computing, in which case a single computing model is difficult to meet the needs of multiple applications, and mashups for multiple computing modes will greatly increase the difficulty of data dumping, using and maintenance. Spark provides a memory computing engine which contains a variety of classic computing models, including streaming computing, batch computing, graph computing and so on. so more and more enterprises choose Spark as a hybrid large data computing platform. In application of Spark Streaming in stream processing, some challenges need to be often faced, which are the characteristics of data flow real-time, dynamic changing and mass and the different delay requirement of multi-user task processing. But, two current dynamic resource allocation strategies are not well to achieve the data flow processing. Aiming at this problem, this paper proposes a dynamic resource allocation model based on multi-applications, which is based on historical data feedback and dynamic adjustment of computing resources to deal with data flow changes and multi-user different requirements. Experimental results show that this method can effectively adjust the resource quota, and reduce processing delay.

Key words: Spark, real-time data stream, multi-applications, dynamic resource allocation