Abstract: Considering poor performance caused by vague definition of small files in Hadoop Distributed File System (HDFS), Cut-off Point via Grey Relational Analysis (Cut-GAR) was presented to find the cut-off point between small files and large files, the relationship among the consumed memory of NameNode (M), speeds of MB of Uploaded Files per Second (MUFS), speeds of MB of Accessed Files per Second (MAFS) and file size was analyzed, the proper file sizes according to the three factors, were set respectively as FM, FMUFS and FMAFS. And then, grey relational analysis was taken to weight impacts of the three factors on file size while file size was treated as evaluated object, and M, MUFS and MAFS were employed as evaluated indexes, therefore the weight of evaluated index and relational degree of index-object were obtained. The outcome that the sum of FM, FMUFS, and FMAFS multiplied by the corresponding index weight was regarded the approximate optimal value of cut-off point. As experiment results demonstrate, Cut-GAR achieves a balance among M, MUFS, and MAFS, which improves the performance of small file processing.

Key words: Hadoop Distributed File System (HDFS), small file, cut-off point, Cut-off Point via Grey Relational Analysis (Cut-GAR), grey relational analysis

摘要： 针对Hadoop分布式文件系统(HDFS)中小文件定义模糊导致HDFS处理小文件性能不佳的问题,提出了一种云存储系统中文件分界点的确定方法——Cut-GAR。该方法分析消耗NameNode内存(M)、文件上传速度(MUFS)、文件读取速度(MAFS)与文件大小之间的关系,得出文件大小三个近似最优值,FM、FMUFS、FMAFS;然后利用灰度关联分析,将M、MUFS、MAFS作为评价指标,文件大小作为评价对象,得到评价指标-评价对象灰色关联度以及评价指标所占权重,将FM、FMUFS、FMAFS与对应权重相乘,得出文件分界点。实验结果表明,Cut-GAR在M、MUFS 和MAFS三者之间取得平衡,可以有效确定文件分界点,提升小文件处理性能。

关键词: Hadoop 分布式文件系统, 小文件, 文件分界点, Cut-GAR, 灰色关联分析