Abstract:

Aiming at the sharp increasing of data on the cloud caused by the development and popularization of cloud native technology as well as the bottlenecks of the technology in performance and stability， a Haystack-based storage system was proposed. With the optimization in service discovery， automatic fault tolerance and caching mechanism， the system is more suitable for cloud native business and meets the growing and high-frequent file storage and read/write requirements of the data acquisition， storage and analysis industries. The object storage model used by the system satisfies the massive file storage with high-frequency reads and writes. A simple and unified application interface is provided for business using the storage system， a file caching strategy is applied to improve the resource utilization， and the rich automated tool chain of Kubernetes is adopted to make this storage system easier to deploy， easier to expand， and more stable than other storage systems. Experimental results indicate that the proposed storage system has a certain performance and stability improvement compared with the current mainstream object storage and file systems in the situation of large-scale fragmented data storage with more reads than writes.

Key words: file system, object storage, cloud computing, container orchestration, cloud native business

摘要：

为应对云原生技术的日益发展与普及伴随的云上数据量的激增及该技术在性能与稳定性等方面所出现的瓶颈，提出了一种基于Haystack的存储系统。该存储系统在服务发现、自动容错与缓存方面进行了优化，更适用于云原生业务，以满足数据采集、存储与分析行业不断增长且频次较高的文件存储与读写需求。该存储系统使用对象存储模型来满足高频海量的文件存储，为使用该存储系统的业务提供简单而统一的应用程序接口，应用了文件缓存策略提升资源利用率，同时利用Kubernetes丰富的自动化工具链使该存储系统比其他存储系统更容易部署和扩展且更稳定。实验结果表明，该存储系统在读多于写的大规模碎片数据存储情境下相比目前主流的对象存储与文件系统均有一定的性能与稳定性提升。

关键词: 文件系统, 对象存储, 云计算, 容器编排, 云原生业务