Design of distributed computing framework for foreign exchange market monitoring

doi:10.11772/j.issn.1001-9081.2019061002

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (1): 173-180.DOI: 10.11772/j.issn.1001-9081.2019061002

• Advanced computing • Previous Articles Next Articles

Design of distributed computing framework for foreign exchange market monitoring

CHENG Wenliang¹, WANG Zhihong², ZHOU Yu¹, GUO Yi^2,3,4, ZHAO Junfeng¹

1. Research and Development 2 Department, China Foreign Exchange Trade System Information Technology(Shanghai) Company Limited, Shanghai 201203, China;
2. School of Computer Science and Engineering, East China University of Science and Technology, Shanghai 200237, China;
3. National Engineering Laboratory for Big Data Distribution and Exchange Technologies(Business Intelligence and Visualization Research Center), Shanghai 200237, China;
4. Shanghai Engineering Research Center of Big Data and Internet Audience, Shanghai 200072, China

Received:2019-06-14 Revised:2019-07-25 Online:2020-01-10 Published:2020-01-17
Contact: 过弋
Supported by:
This work is partially supported by the National Key Research and Development Program (2018YFC0807105), the Scientific Research Project of the Science and Technology Committee of Shanghai Municipality (17DZ1101003, 18511106602, 18DZ2252300).

面向外汇市场监测的分布式计算框架设计

程文亮¹, 王志宏², 周虞¹, 过弋^2,3,4, 赵俊锋¹

1. 中汇信息技术(上海)有限公司开发二部, 上海 201203;
2. 华东理工大学信息科学与工程学院, 上海 200237;
3. 大数据流通与交易技术国家工程实验室(商业智能与可视化研究中心), 上海 200237;
4. 上海大数据与互联网受众工程技术研究中心, 上海 200072

作者简介:程文亮(1989-),男,湖北天门人,工程师,硕士,主要研究方向:电子交易系统设计与开发;王志宏(1990-),男,江苏泰兴人,博士研究生,CCF会员,主要研究方向:文本挖掘;周虞(1981-),男,浙江定海人,高级工程师,主要研究方向:电子交易系统设计与开发;过弋(1975-),男,江苏无锡人,教授,博士生导师,博士,CCF高级会员,主要研究方向:文本挖掘、知识发现、商业智能;赵俊锋(1979-),男,广东江门人,高级工程师,硕士,主要研究方向:电子交易系统设计与开发。
基金资助:
国家重点研发计划项目（2018YFC0807105）；上海市科学技术委员会科研计划项目（17DZ1101003，18511106602，18DZ2252300）。

Abstract

Abstract: In order to solve the index calculation problems of high complexity, strong completeness and low efficiency in the filed of financial foreign exchange market monitoring, a novel distributed computing framework for foreign exchange market monitoring based on Spark big data structure was proposed. Firstly, the business characteristics and existing technology framework for foreign exchange market monitoring were analyzed and summarized. Secondly, the foreign exchange business features of single-market multi-indicator and multi-market multi-indicator were considered. Finally, based on Spark's Directed Acyclic Graph (DAG) job scheduling mechanism and resource scheduling pool isolation mechanism of YARN (Yet Another Recourse Negotiator), the Market-level DAG (M-DAG) model and the market-level resource allocation strategy named M-YARN (Market-level YARN) model were proposed, respectively. The experimental results show that, the performance of the proposed distributed computing framework for foreign exchange market monitoring improves the performance by more than 80% compared to the traditional technology framework, and can effectively guarantee the completeness, accuracy and timeliness of foreign exchange market monitoring indicator calculation under the background of big data.

Key words: foreign exchange market, market monitoring, Spark, Directed Acyclic Graph (DAG), resource allocation

摘要： 针对金融外汇市场监测指标计算复杂度高、完备性强、效率低等问题，基于Spark大数据架构提出了一种新的面向外汇市场监测的分布式计算框架。首先，对外汇市场监测的业务特性和现有技术框架进行了分析总结；然后，综合考虑了外汇单市场多指标和多市场多指标并行计算的业务特性；最后，基于Spark的有向无环图（DAG）作业调度机制和YARN的资源调度池隔离机制，分别提出了外汇市场级的有向无环图（M-DAG）模型和市场级资源分配策略——M-YARN。实验结果表明，所提面向外汇市场监测的分布式计算框架相对于传统技术框架在性能上提高了80%以上，可以有效保证大数据背景下外汇市场监测指标计算的完备性、精准性和时效性。

关键词: 外汇市场, 市场监测, Spark, 有限无环图, 资源分配

CLC Number:

TP399

CHENG Wenliang, WANG Zhihong, ZHOU Yu, GUO Yi, ZHAO Junfeng. Design of distributed computing framework for foreign exchange market monitoring[J]. Journal of Computer Applications, 2020, 40(1): 173-180.

程文亮, 王志宏, 周虞, 过弋, 赵俊锋. 面向外汇市场监测的分布式计算框架设计[J]. 计算机应用, 2020, 40(1): 173-180.

References

[1] Wikipedia. Foreign exchange market[EB/OL].[2019-03-20].https://en.wikipedia.org/wiki/Foreign_exchange_market.
[2] 么贻聪.浅谈监管科技在外汇管理领域的应用[J].吉林金融研究,2018(5):72-74.(YAO Y C. Discussion on the application of regtechin foreign exchange management[J]. Journal of Jilin Financial Research, 2018(5):72-74.)
[3] 刘芳,薛蕾.利用数据挖掘技术监测外汇公有账户资金流失[J].计算机工程与应用,2006,42(12):227-228,232.(LIU F, XUE L. Using data mining techniques to monitor the leakage of money from corporation foreign exchange accounts[J]. Computer Engineering and Applications, 2006, 42(12):227-22.)
[4] 孙景,张成虎,陈善新.基于时间序列孤立点检测的可疑外汇资金交易识别研究[J].统计与决策,2010(18):26-29.(SUN J, ZHANG C H, CHEN S X. Research on suspicious foreign exchange fund transaction identification based on time series isolated point detection[J]. Statistics and Decision, 2010(18):26-29.)
[5] 郑义平.数据综合利用在外汇管理中的应用实践[J].金融科技时代,2018(5):67-69.(ZHENG Y P. Application practice of data comprehensive utilization in foreign exchange management[J]. Financial Technology Time, 2018(5):67-69.)
[6] 刘祺,王丽娜.构建全口径个人外汇业务监测体系[J].中国外汇,2016(7):83-83.(LIU Q, WANG L N. Construction of a full-scale personal foreign exchange business monitoring system[J]. China Forex, 2016(7):83-83.)
[7] 钟庆作,温路娟.完善个人外汇业务监测系统[J].中国外汇,2018(5):75-75.(ZHONG Q Z, WEN L J. Improving personal foreign exchange business monitoring system[J]. China Forex, 2018(5):75-75.)
[8] 李阳.个人外汇业务监测系统运行中出现的新问题及建议[J].黑龙江金融,2016(8):37-39.(LI Y. New problems and suggestions in the operation of personal foreign exchange business monitoring system[J]. Heilongjiang Finance, 2016(8):37-39.)
[9] 徐珊,李慧强.对大数据分析在外汇管理领域运用的思考[J].金融纵横,2017(9):87-92.(XU S, LI H Q. Thoughts on the application of big data analysis in foreign exchange management[J]. Financial Perspectives Journal, 2017(9):87-92.)
[10] 郑义平.数据综合利用在外汇管理中的应用实践[J].金融科技时代,2018(5):70-72.(ZHENG Y P. Application practice of data comprehensive utilization in foreign exchange management[J]. Financial Technology Time, 2018(5):70-72.)
[11] 吴水平,陈晓虹.构建金融市场监测数据库[J].中国金融,2017(8):87-88.(WU S P, CHEN X H. Building a financial market monitoring database[J]. China Finance, 2017(8):87-88.)
[12] 曲朝阳,陈贺新,胡可为,等.基于Spark的电力调度数据整合模型[J].计算机工程与应用,2017,53(19):65-70.(QU Z Y, CHEN H X, HU K W, et al. Power dispatch data integration model based on Spark[J]. Computer Engineering and Applications, 2017, 53(19):65-70.)
[13] 盛剑,樊红,龚天任,等.K-Means算法及其在卷烟零售门店库存聚类分析中的应用[J].商业经济,2017(3):128-129.(SHENG J, FAN H, GONG T R, et al. K-Means algorithm and the application of inventory clustering analysis in the cigarette retail stores[J]. Business Economy, 2017(3):128-129.)
[14] 陈萌,赵焕芳,杨声钢,等.基于Spark计算框架的担保能力计量模型[J].中国金融电脑,2017(3):52-55.(CHEN M, ZHAO H F, YANG S G, et al. Eligibility capability measurement model based on Spark[J]. Financial Computer of China, 2017(3):52-55.)
[15] LIU J. Sparks of the CRISPR explosion:Applications in medicine and agriculture[J]. Journal of Genetics and Genomics, 2017, 44(9):413-414.
[16] HUA Y, PAN J, YAN Z, et al. An empirical comparison of three ensemble methods for medical data mining with Apache Spark[C]//Proceedings of the IEEE 12th International Conference on Ubiquitous Intelligence and Computing and IEEE 12th International Conference on Autonomic and Trusted Computing and IEEE 15th International Conference on Scalable Computing and Communications and Its Associated Workshop. Piscataway:IEEE, 2017:917-922.
[17] 廖湖声,黄珊珊,徐俊刚,等.Spark性能优化技术研究综述[J].计算机科学,2018,45(7):7-15,37.(LIAO H S, HUANG S S, XU J G, et al. Survey on performance optimization technologies for Spark[J]. Computer Science, 2018, 45(7):7-15, 37.)
[18] 易纲.全面提升外汇监管和服务水平[J].中国金融,2010(5):8-9.(YI G. To comprehensively improve foreign exchange supervision and service level[J]. China Finance, 2010(5):8-9.)
[19] ORACLE. Oracle financial services software limited:annual report 2017-2018[EB/OL].[2019-02-22]. http://www.oracle.com/us/industries/financial-services/ofss-annual-report-2017-18-5019118.pdf.
[20] BARR E T, HARMAN M, MCMINN P, et al. The Oracle problem in software testing:a survey[J]. IEEE Transactions on Software Engineering, 2015, 41(5):507-525.
[21] LIU L, YANG H, WESMMIAN A. Research on the enterprise resource management system and auxiliary performance evaluation mechanism based on oracle database optimization and security enhancement[J]. International Journal of Future Generation Communication and Networking, 2017, 10(1):135-148.
[22] 殷荣.基于DAG模型的离线数据处理引擎的设计与实现[D].哈尔滨:哈尔滨工业大学,2016:18-20.(YIN R. The design and implementation of offline data processing engine based on DAG model[D]. Harbin:Harbin Institute of Technology, 2016:18-20.)
[23] 王荣丽,侯秀萍.基于优先级权重的Hadoop YARN调度算法[J].吉林大学学报(信息科学版),2017,35(4):443-448.(WANG R L, HOU X P. Hadoop YARN scheduling algorithm based on priority weight[J]. Journal of Jilin University (Information Science Edition), 2017, 35(4):443-448.)
[24] LI W, LIU X, ZHANG X, et al. A further analysis of the dynamic dominant resource fairness mechanism[C]//Proceedings of the 2017 International Workshop on Frontiers in Algorithmics, LNCS 10336. Cham:Spring, 2017:163-174.
[25] 班子寒,张阳.BH-Quant智能量化策略辅助设计平台的研究与实践[J].软件工程,2017,20(9):1-5.(BAN Z H, ZHANG Y. Research and practice of the BH-Quant intelligent quantization strategy aided design platform[J]. Software Engineer, 2017, 20(9):1-5.)

Design of distributed computing framework for foreign exchange market monitoring

面向外汇市场监测的分布式计算框架设计

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	JIANG Kun, LIU Zheng, ZHU Lei, LI Xiaoxing. Fixed word-aligned partition compression algorithm of inverted list based on directed acyclic graph [J]. Journal of Computer Applications, 2021, 41(3): 727-732.
[2]	HUANG Dongyan, FU Zhongwei, WANG Bo. Revenue maximization strategy for mobile-edge computing server with limited computing resources [J]. Journal of Computer Applications, 2020, 40(3): 765-769.
[3]	GU Junhua, WANG Feng, QI Yongjun, SUN Zheran, TIAN Zepei, ZHANG Yajuan. Retrieval method of pulmonary nodule images based on multi-scale convolution feature fusion [J]. Journal of Computer Applications, 2020, 40(2): 561-565.
[4]	ZHANG Xiajie, ZHU Jinghua, CHEN Yang. Distributed rough set attribute reduction algorithm under Spark [J]. Journal of Computer Applications, 2020, 40(2): 518-523.
[5]	DU Ming, YANG Anping, ZHOU Junfeng, CHEN Ziyang, YANG Yun. Optimized algorithm for k-step reachability queries on directed acyclic graphs [J]. Journal of Computer Applications, 2020, 40(2): 426-433.
[6]	LIU Bin, HE Jinrong, LI Yuancheng, HAN Hong. Apple price prediction method based on distributed neural network [J]. Journal of Computer Applications, 2020, 40(2): 369-374.
[7]	HUANG Xiuli, HUANG Jin, YU Pengfei, MIAO Weiwei, YANG Ruxia, LI Yijing, YU Peng. Security-risk-oriented distributed resource allocation method in power wireless private network [J]. Journal of Computer Applications, 2020, 40(12): 3586-3593.
[8]	CUI Yixin, CHEN Xiaodong. Spark framework based optimized large-scale spectral clustering parallel algorithm [J]. Journal of Computer Applications, 2020, 40(1): 168-172.
[9]	AN Xin, ZHANG Ying, KANG An, CHEN Tian, LI Jianhua. Machine learning based online mapping approach for heterogeneous multi-core processor system [J]. Journal of Computer Applications, 2019, 39(6): 1753-1759.
[10]	XU Yabin, PENG Hong'en. PaaS platform resource allocation method based on demand forecasting [J]. Journal of Computer Applications, 2019, 39(6): 1583-1588.
[11]	LIU Jing, XIAO Guanfeng. Bus arrival time prediction system based on Spark and particle filter algorithm [J]. Journal of Computer Applications, 2019, 39(2): 429-435.
[12]	LIU Zihao, LI Ling, YE Feng. Anomaly detection method for hydrologic sensor data based on SparkR [J]. Journal of Computer Applications, 2019, 39(2): 436-440.
[13]	YU Yun, LIAN Xiaocan, ZHU Yuhang, TAN Guoping. Resource allocation optimization method for augment reality applications based on mobile edge computing [J]. Journal of Computer Applications, 2019, 39(1): 22-25.
[14]	LI Longyang, DONG Yihong, SHI Weijie, PAN Jianfei. SQM: subgraph matching algorithm for single large-scale graphs under Spark [J]. Journal of Computer Applications, 2019, 39(1): 46-50.
[15]	ZHAO Wenfang, WANG Jingli, SHANG Min, LIU Yanan. Forecasting model of pollen concentration based on particle swarm optimization and support vector machine [J]. Journal of Computer Applications, 2019, 39(1): 98-104.