Journal of Computer Applications ›› 2018, Vol. 38 ›› Issue (6): 1596-1600.DOI: 10.11772/j.issn.1001-9081.2017112632

Previous Articles     Next Articles

Efficient block-based sampling algorithm for aggregation query processing on duplicate charged records

PAN Mingyu1, ZHANG Lu1, LONG Guobiao1, LI Xianglong1, MA Dongxue1, XU Liang2   

  1. 1. State Grid Beijing Electric Power Company, Beijing 100075, China;
    2. NARI Group, Beijing 102299, China
  • Received:2017-11-06 Revised:2018-02-05 Online:2018-06-10 Published:2018-06-13
  • Supported by:
    This work is partially supported by the Science and Technology Project of State Grid Corporation (52020116000j).


潘鸣宇1, 张禄1, 龙国标1, 李香龙1, 马冬雪1, 徐亮2   

  1. 1. 国网北京市电力公司, 北京 100075;
    2. 南瑞集团, 北京 102299
  • 通讯作者: 潘鸣宇
  • 作者简介:潘鸣宇(1985-),男,北京人,高级工程师,硕士,主要研究方向:充换电运营数据分析;张禄(1984-),男,北京人,高级工程师,博士,主要研究方向:充换电运营数据分析;龙国标(1967-),男,北京人,高级工程师,主要研究方向:充换电运营数据分析;李香龙(1980-),男,河北石家庄人,高级工程师,硕士,主要研究方向:充换电运营数据分析;马冬雪(1989-),女,北京人,中级经济师,硕士,主要研究方向:充换电运营咨询;徐亮(1981-),男,辽宁沈阳人,高级工程师,硕士,主要研究方向:充换电运营数据分析。
  • 基金资助:

Abstract: The existing query analysis methods usually treat the entity resolution as an offline preprocessing process to clean the whole data set. However, with the continuous increasing of data size, such offline cleaning mode with high computing complexity has been difficult to meet the needs of real-time analysis in most applications. In order to solve the problem of aggregation query on duplicate charged records, a new method integrating entity resolution with approximate aggregation query processing was proposed. Firstly, a block-based sampling strategy was adopted to collect samples. Then, an entity recognition method was used to identify the duplicate entities on the sampled samples. Finally, the unbiased estimation of aggregated results was reconstructed according to the results of entity recognition. The proposed method avoids the time cost of identifying all entities, and returns the query results that satisfy user needs by identifying only a small number of sample data. The experimental results on both real dataset and synthetic dataset demonstrate the efficiency and reliability of the proposed method.

Key words: big data, entity resolution, aggregation query, block sampling, distributed computing

摘要: 现有查询分析方法通常将实体识别作为线下预处理过程清洗整个数据集,然而,随着数据规模的不断增大,这种高计算复杂性的线下清洗模式已经很难满足实时性分析应用的需求。针对重复充电运营记录上的聚集查询问题,提出一种将近似聚集查询处理与实体识别相结合的方法。首先,通过基于块的采样策略采集样本;然后,在采集到的样本上利用实体识别方法识别出重复的实体;最后,根据实体识别的结果重构得到聚集结果的无偏估计。所提方法避免了识别全部实体的时间代价,通过识别少量样本数据即可返回满足用户需求的查询结果。真实数据集和合成数据集上的实验结果验证了所提方法的高效性和可靠性。

关键词: 大数据, 实体识别, 聚集查询, 块采样, 分布式计算

CLC Number: