Real-time processing of space science satellite data based on stream computing

doi:10.11772/j.issn.1001-9081.2018122602

Journal of Computer Applications ›› 2019, Vol. 39 ›› Issue (6): 1563-1568.DOI: 10.11772/j.issn.1001-9081.2018122602

• 2018 National Annual Conference on High Performance Computing (HPC China 2018) • Previous Articles Next Articles

Real-time processing of space science satellite data based on stream computing

SUN Xiaojuan^1,2,3, SHI Tao^1,2, HU Yuxin^1,2,3, TONG Jizhou^3,4, LI Bing^1,2, SONG Yao^1,2,3

1. Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China;
2. Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Chinese Academy of Sciences, Beijing 100190, China;
3. University of Chinese Academy of Sciences, Beijing 100049, China;
4. National Space Science Center, Chinese Academy of Sciences, Beijing 100190, China

Received:2018-12-12 Revised:2019-02-03 Online:2019-06-10 Published:2019-06-17
Supported by:
This work is partially supported by the 13th Five-year Information Plan of Chinese Academy of Sciences (XXH13505-04), the Beijing Science and Technology Plan (Z181100002918002).

基于流式计算的空间科学卫星数据实时处理

孙小涓^1,2,3, 石涛^1,2, 胡玉新^1,2,3, 佟继周^3,4, 李冰^1,2, 宋峣^1,2,3

1. 中国科学院电子学研究所, 北京 100190;
2. 中国科学院空间信息处理与应用系统技术重点实验室, 北京 100190;
3. 中国科学院大学, 北京 100049;
4. 中国科学院国家空间科学中心, 北京 100190

通讯作者: 孙小涓
作者简介:孙小涓(1980-),女,山东烟台人,副研究员,博士,CCF会员,主要研究方向:空间信息处理、高性能计算;石涛(1982-),男,天津人,副研究员,硕士,CCF会员,主要研究方向:空间信息处理;胡玉新(1981-),男,内蒙古赤峰人,研究员,博士,主要研究方向:空间信息处理系统;佟继周(1976-),女,北京人,副研究员,硕士,主要研究方向:空间科学大数据管理;李冰(1990-),男,山西运城人,助理研究员,硕士,主要研究方向:空间信息处理;宋峣(1994-),男,江西新余人,硕士,主要研究方向:空间信息处理。
基金资助:
中国科学院十三五信息化专项（XXH13505-04）；北京市科技计划项目（Z181100002918002）。

Abstract

Abstract: Concerning the increasingly high real-time processing requirement of space science satellite observed data, a real-time processing method of space science satellite data based on stream computing framework was proposed. Firstly, the data stream was abstractly analyzed according to the data processing characteristics of space science satellite. Then, the input and output data structures of each processing unit were redefined. Finally, the parallel data stream processing structure was designed based on the stream computing framework Storm to meet the requirements of parallel processing and distributed computing of large-scale data. The developed system for space science satellite data processing applying with this method was tested and analyzed. The results show that the data processing time is half of that of the original system under same conditions and the data localization strategy has higher throughput than round-robin strategy with the data tuple throughput increased by 29% on average. It can be seen that the use of stream computing framework can greatly shorten the data processing delay and improve the real-time performance of the space science satellite data processing system.

Key words: stream computing, data stream, Storm, space science satellite, data processing

摘要： 针对空间科学卫星探测数据的实时处理要求越来越高的问题，提出一种基于流计算框架的空间科学卫星数据实时处理方法。首先，根据空间科学卫星数据处理特点对数据流进行抽象分析；然后，对各处理单元的输入输出数据结构进行重新定义；最后，基于流计算框架Storm设计数据流处理并行结构，以适应大规模数据并行处理和分布式计算的要求。对应用该方法开发的空间科学卫星数据处理系统进行测试分析，测试结果显示，在相同条件下数据处理时间比原有系统缩短了一半；数据局部性策略比轮询策略具有更高的吞吐率，数据元组吞吐率平均提高29%。可见采用流式计算框架能够大幅缩短数据处理延迟，提高空间科学卫星数据处理系统的实时性。

关键词: 流式计算, 数据流, Storm, 空间科学卫星, 数据处理

CLC Number:

TP319

SUN Xiaojuan, SHI Tao, HU Yuxin, TONG Jizhou, LI Bing, SONG Yao. Real-time processing of space science satellite data based on stream computing[J]. Journal of Computer Applications, 2019, 39(6): 1563-1568.

孙小涓, 石涛, 胡玉新, 佟继周, 李冰, 宋峣. 基于流式计算的空间科学卫星数据实时处理[J]. 计算机应用, 2019, 39(6): 1563-1568.

References

[1] 中国科学院国家空间科学中心.空间科学任务运行网[EB/OL].(2015-12-31)[2018-07-31]. http://www.smoc.ac.cn/. (National Space Science Center, Chinese Academy of Sciences. Space science mission operation network[EB/OL]. (2015-12-31)[2018-07-31]. http://www.smoc.ac.cn/.)
[2] 中国科学院.暗物质粒子探测卫星工程[EB/OL].(2015-12-31)[2018-07-31].http://www.cas.cn/zt/kjzt/awzlztcwxgc/.(Chinese Academy of Sciences. DAMPE satellite engineering[EB/OL]. (2015-12-31)[2018-07-31]. http://www.cas.cn/zt/kjzt/awzlztcwxgc/.)
[3] 中国科学院.实践十号返回式科学实验卫星工程[EB/OL].(2016-12-31)[2018-07-31].http://www.cas.cn/zt/kjzt/sjshkxsywx/.(Chinese Academy of Sciences. SJ-10 recoverable scientific experiment satellite engineering[EB/OL]. (2016-12-31)[2018-07-31]. http://www.cas.cn/zt/kjzt/sjshkxsywx/.)
[4] 中国科学院.量子科学实验卫星工程[EB/OL].(2016-12-31)[2018-07-31].http://www.cas.cn/zt/kjzt/lzwx/.(Chinese Academy of Sciences. QUESS satellite engineering[EB/OL]. (2016-12-31)[2018-07-31]. http://www.cas.cn/zt/kjzt/lzwx/.)
[5] 中国科学院.硬X调制望远镜卫星[EB/OL].(2017-12-31)[2018-07-31].http://www.cas.cn/zt/kjzt/yxsxtzwyj/.(Chinese Academy of Sciences. HXMT satellite[EB/OL]. (2017-12-31)[2018-07-31]. http://www.cas.cn/zt/kjzt/yxsxtzwyj/.)
[6] ABBOTT B P, ABBOTT R, ABBOTT T D, et al. GW170817:observation of gravitational waves from a binary neutron star inspiral[J]. Physical Review Letters, 2017, 119(16):161101.
[7] The Gamma-ray Coordinates Network. Insight-HXMT observation[EB/OL]. (2018-02-05)[2018-07-31]. https://gcn.gsfc.nasa.gov/gcn3/21593.gcn3/.
[8] 孙小涓,石涛,李冰,等.空间科学卫星快速数据处理方法[C]//2017全国高性能计算学术年会论文集.北京:中国计算机学会,2017:438-443.(SUN X J, SHI T, LI B, et al. A rapid data processing method for space science satellites[C]//Proceedings of the 2017 HPC China. Beijing:CCF, 2017:438-443.)
[9] 孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.(SUN D W, ZHANG G Y, ZHENG W M. Big data stream computing:technologies and instances[J]. Journal of Software, 2014, 25(4):839-862.)
[10] 卫守林,刘鹏翔,王锋,等.基于Spark Streaming的明安图射电频谱日像仪实时数据处理[J].天文研究与技术,2017,14(4):421-428.(WEI S L, LIU P X, WANG F, et al. Real-time data processing in Mingantu ultrawide spectral radio heliograph based on Spark Streaming[J]. Astronomical Research and Technology, 2017, 14(4):421-428.)
[11] SIMONCELLI D, DUSI M, GRINGOLI F,et al. Scaling out the performance of service monitoring applications with BlockMon[C]//Proceedings of the 2013 International Conference on Passive and Active Measurement, LNCS 7799. Berlin:Springer, 2013:253-255.
[12] 韩德志,陈旭光,雷雨馨,等.基于Spark Streaming的实时数据分析系统及其应用[J].计算机应用,2017,37(5):1263-1269.(HAN D Z, CHEN X G, LEI Y X, et al. Real-time data analysis system based on Spark Streaming and its application[J]. Journal of Computer Applications, 2017, 37(5):1263-1269.)
[13] NEUMEYER L, ROBBINS B, KESARI A, et al. S4:distributed stream computing platform[C]//Proceedings of the 2010 IEEE International Conference on Data Mining Workshops. Washington, DC:IEEE Computer Society, 2010:170-177.
[14] LI F, DAI L, JIANG Z, et al. Single-pass clustering algorithm based on Storm[J]. Journal of Physics:Conference Series, 2017, 806:012017.
[15] 丁维龙,赵卓峰,韩燕波.Storm:大数据流式计算及应用实践[M].北京:电子工业出版社,2015:110-137.(DING W L, ZHAO Z F, HAN Y B. Storm:Big Data Stream Computing and Application Practice[M]. Beijing:Publishing House of Electronics Industry, 2015:110-137.)
[16] CHINTAPALLI S, DAGIT D, EVANS R, et al. PaceMaker:when ZooKeeper arteries get clogged in storm clusters[C]//Proceedings of the 9th IEEE International Conference on Cloud Computing. Piscataway, NJ:IEEE, 2017:448-455.
[17] KARUNARATNE P, KARUNASEKERA S, HARWOOD A. Distributed stream clustering using micro-clusters on Apache Storm[J]. Journal of Parallel & Distributed Computing, 2017, 108:74-84.
[18] NASA. Space packet protocol:CCSDS 133.0-B-1[S]. Washington, DC:CCSDS, 2003-09.
[19] 马苗,朱岩.基于CCSDS标准的卫星数据处理软件设计[J].电子设计工程,2015,23(1):16-20.(MA M, ZHU Y. Design of satellite data processing software based on CCSDS[J]. Electronic Design Engineering, 2015, 23(1):16-20.)

Real-time processing of space science satellite data based on stream computing

基于流式计算的空间科学卫星数据实时处理

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	LI Mengmeng, QIN Wei, LIU Yi, DIAO Xingchun. Hybrid ant colony optimization algorithm with brain storm optimization [J]. Journal of Computer Applications, 2021, 41(8): 2412-2417.
[2]	YIN Chunyong, ZHANG Guojie. Ensemble classification model for distributed drifted data streams [J]. Journal of Computer Applications, 2021, 41(7): 1947-1955.
[3]	GUO Shuai, SU Yang. Encrypted traffic classification method based on data stream [J]. Journal of Computer Applications, 2021, 41(5): 1386-1391.
[4]	LI Lirong, YANG Kun, WANG Peichong. Improved teaching & learning based optimization with brain storming [J]. Journal of Computer Applications, 2020, 40(9): 2677-2682.
[5]	ZHANG Chenxi, TANG Shu, TANG Ke. Strategies of parameter fault detection for rocket engines based on transfer learning [J]. Journal of Computer Applications, 2020, 40(9): 2774-2780.
[6]	FAN Zhongxin. Clustering tendency analysis algorithm based on data stream [J]. Journal of Computer Applications, 2020, 40(8): 2248-2254.
[7]	ZHU Xiaojie, ZHAO Zihao, DU Yi. PiFlow: model driven big data pipeline framework [J]. Journal of Computer Applications, 2020, 40(6): 1638-1647.
[8]	WANG Ning, LIU Yong. Adaptive most valuable player algorithm considering multiple training methods [J]. Journal of Computer Applications, 2020, 40(6): 1722-1730.
[9]	ZHANG Yitian, YU Jiong, LU Liang, LI Ziyang. Task scheduling strategy based on data stream classification in Heron [J]. Journal of Computer Applications, 2019, 39(4): 1106-1116.
[10]	HAN Meng, DING Jian. Survey of frequent pattern mining over data streams [J]. Journal of Computer Applications, 2019, 39(3): 719-727.
[11]	LIU Jing, XIAO Guanfeng. Bus arrival time prediction system based on Spark and particle filter algorithm [J]. Journal of Computer Applications, 2019, 39(2): 429-435.
[12]	LIANG Zhigang, GU Junhua. Medical image registration by integrating modified brain storm optimization algorithm and Powell algorithm [J]. Journal of Computer Applications, 2018, 38(9): 2683-2688.
[13]	LI Ziyang, YU Jiong, BIAN Chen, LU Liang, PU Yonglin. Dynamic task dispatching strategy for stream processing based on flow network [J]. Journal of Computer Applications, 2018, 38(9): 2560-2567.
[14]	SU Leihao, ZHU Minghua. Vibration detection system based on process field bus technology [J]. Journal of Computer Applications, 2018, 38(7): 2113-2118.
[15]	YUAN Quan, GUO Jiangfan. New ensemble classification algorithm for data stream with noise [J]. Journal of Computer Applications, 2018, 38(6): 1591-1595.