关键词: 关键词: 大数据分类, 分布式文件系统, 随机样本划分, 观测点分类器, Spark计算框架

Abstract: Abstract: Observation point classifier (OPC) is a latest supervised learning model which tries to transform the multi-dimensional linearly-inseparable problem in the original data space into the one-dimensional linearly-separable problem in the projective distance space. In order to alleviate the high computational complexity when applying OPC to handle the big data classification problem, a random sample partition (RSP)-based distributed OPC (DOPC) for big data was designed under the Spark framework to enhance the efficiency of OPC to classify the big data. First, RSP data blocks were generated and transformed into RDD under the distributed computation environment. Second, a series of OPC was synergistically trained based on RSP data blocks with high Spark parallelizability. Finally, the predictive results of different OPC were fused into the final label of unknow sample. The persuasive experiments based on 8 big data sets were conducted to validate the feasibility, rationality and effectiveness of designed DOPC. Experimental results show that DOPC trained on multiple computation nodes gets the better generalization capability than OPC trained on single computation node with less time consumption and meanwhile compared to the RSP model based neural network, decision tree, na?ve Bayesian, and K-nearest neighbor classifiers under the Spark framework, the average testing accuracy of DOPC has been improved by 1.7%, 0.2%, 12.1%, and 1.9%, respectively. The superior testing performances demonstrate that DOPC is a high-efficiency and low-consumption supervised learning algorithm for handling big data classification problems.

Key words: Abstract: Big data classification, Distributed file system, Random sample partition, Observation point classifier, Spark computing framework