关键词: 关键词: 气象自动站, spark streaming, 流计算, 数据处理, flume

Abstract: Abstract: Aiming at these problems of the current data service of Automatic Weather Stations (AWS), including data processing delay, slow interactive response, and low statistical efficiency, a new method based on Spark Streaming and HBase technologies was proposed and introduced for processing massive streaming AWS data by integrating stream computing framework and distributed database system. The method uses Flume for data collection, Spark Streaming for data processing and data storage into HBase. In framework of Spark, two algorithms, one for writing streaming AWS data into HBase database, the other for realizing real-time statistical calculation of different observed AWS meteorological elements were designed. Finally, a stable and high-efficiency system for real-time acquisition, processing, and statistics of AWS data is developed on Cloudera platform. Based on comparative analysis and running monitoring, robust performances of the system have been confirmed, including low latency time, high I/O efficiency, stable running status, and excellent load balance. The test results show that response time of Spark Streaming-based real-time operational processing of AWS data can reach to millisecond level, which includes paralleled data writing into HBase, HBase-based data query and statistics on different meteorological elements. The system can fully meet needs of operational applications on AWS data, and provide effective support to weather forecast.

Key words: Keywords: automated weather station, spark streaming, streaming computer, data processing, flume