计算机应用 ›› 2020, Vol. 40 ›› Issue (1): 173-180.DOI: 10.11772/j.issn.1001-9081.2019061002

• 先进计算 • 上一篇    下一篇

面向外汇市场监测的分布式计算框架设计

程文亮1, 王志宏2, 周虞1, 过弋2,3,4, 赵俊锋1   

  1. 1. 中汇信息技术(上海)有限公司 开发二部, 上海 201203;
    2. 华东理工大学 信息科学与工程学院, 上海 200237;
    3. 大数据流通与交易技术国家工程实验室(商业智能与可视化研究中心), 上海 200237;
    4. 上海大数据与互联网受众工程技术研究中心, 上海 200072
  • 收稿日期:2019-06-14 修回日期:2019-07-25 出版日期:2020-01-10 发布日期:2020-01-17
  • 作者简介:程文亮(1989-),男,湖北天门人,工程师,硕士,主要研究方向:电子交易系统设计与开发;王志宏(1990-),男,江苏泰兴人,博士研究生,CCF会员,主要研究方向:文本挖掘;周虞(1981-),男,浙江定海人,高级工程师,主要研究方向:电子交易系统设计与开发;过弋(1975-),男,江苏无锡人,教授,博士生导师,博士,CCF高级会员,主要研究方向:文本挖掘、知识发现、商业智能;赵俊锋(1979-),男,广东江门人,高级工程师,硕士,主要研究方向:电子交易系统设计与开发。
  • 基金资助:
    国家重点研发计划项目(2018YFC0807105);上海市科学技术委员会科研计划项目(17DZ1101003,18511106602,18DZ2252300)。

Design of distributed computing framework for foreign exchange market monitoring

CHENG Wenliang1, WANG Zhihong2, ZHOU Yu1, GUO Yi2,3,4, ZHAO Junfeng1   

  1. 1. Research and Development 2 Department, China Foreign Exchange Trade System Information Technology(Shanghai) Company Limited, Shanghai 201203, China;
    2. School of Computer Science and Engineering, East China University of Science and Technology, Shanghai 200237, China;
    3. National Engineering Laboratory for Big Data Distribution and Exchange Technologies(Business Intelligence and Visualization Research Center), Shanghai 200237, China;
    4. Shanghai Engineering Research Center of Big Data and Internet Audience, Shanghai 200072, China
  • Received:2019-06-14 Revised:2019-07-25 Online:2020-01-10 Published:2020-01-17
  • Contact: 过弋
  • Supported by:
    This work is partially supported by the National Key Research and Development Program (2018YFC0807105), the Scientific Research Project of the Science and Technology Committee of Shanghai Municipality (17DZ1101003, 18511106602, 18DZ2252300).

摘要: 针对金融外汇市场监测指标计算复杂度高、完备性强、效率低等问题,基于Spark大数据架构提出了一种新的面向外汇市场监测的分布式计算框架。首先,对外汇市场监测的业务特性和现有技术框架进行了分析总结;然后,综合考虑了外汇单市场多指标和多市场多指标并行计算的业务特性;最后,基于Spark的有向无环图(DAG)作业调度机制和YARN的资源调度池隔离机制,分别提出了外汇市场级的有向无环图(M-DAG)模型和市场级资源分配策略——M-YARN。实验结果表明,所提面向外汇市场监测的分布式计算框架相对于传统技术框架在性能上提高了80%以上,可以有效保证大数据背景下外汇市场监测指标计算的完备性、精准性和时效性。

关键词: 外汇市场, 市场监测, Spark, 有限无环图, 资源分配

Abstract: In order to solve the index calculation problems of high complexity, strong completeness and low efficiency in the filed of financial foreign exchange market monitoring, a novel distributed computing framework for foreign exchange market monitoring based on Spark big data structure was proposed. Firstly, the business characteristics and existing technology framework for foreign exchange market monitoring were analyzed and summarized. Secondly, the foreign exchange business features of single-market multi-indicator and multi-market multi-indicator were considered. Finally, based on Spark's Directed Acyclic Graph (DAG) job scheduling mechanism and resource scheduling pool isolation mechanism of YARN (Yet Another Recourse Negotiator), the Market-level DAG (M-DAG) model and the market-level resource allocation strategy named M-YARN (Market-level YARN) model were proposed, respectively. The experimental results show that, the performance of the proposed distributed computing framework for foreign exchange market monitoring improves the performance by more than 80% compared to the traditional technology framework, and can effectively guarantee the completeness, accuracy and timeliness of foreign exchange market monitoring indicator calculation under the background of big data.

Key words: foreign exchange market, market monitoring, Spark, Directed Acyclic Graph (DAG), resource allocation

中图分类号: