计算机应用 ›› 2012, Vol. 32 ›› Issue (06): 1724-1726.DOI: 10.3724/SP.J.1087.2012.01724

• 计算机软件技术 • 上一篇    下一篇

基于Hadoop的海量MP3文件存储架构研究

赵晓永,杨扬,孙莉莉,陈宇   

  1. 北京科技大学 计算机与通信工程学院, 北京 100083
  • 收稿日期:2011-11-28 修回日期:2012-02-03 发布日期:2012-06-04 出版日期:2012-06-01
  • 通讯作者: 赵晓永
  • 作者简介:赵晓永(1981-),男,河南郑州人,博士研究生,主要研究方向:云存储;〓杨扬(1955-),男,河北承德人,教授,博士生导师,CCF会员,主要研究方向:云计算、网格计算;〓孙莉莉(1986-),女,黑龙江绥化人,博士研究生,主要研究方向:云计算、虚拟化;〓陈宇(1989-),男,北京人,硕士研究生,主要研究方向:云计算。
  • 基金资助:
    国家自然科学基金资助项目

Hadoop-based storage architecture for mass MP3 files

ZHAO Xiao-yong,YANG Yang,SUN Li-li,CHEN Yu   

  1. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China
  • Received:2011-11-28 Revised:2012-02-03 Online:2012-06-04 Published:2012-06-01
  • Contact: ZHAO Xiao-yong

摘要: MP3作为数字音乐时代的事实标准,数量已相当巨大,用户的访问需求也与日俱增,有效存储和管理海量的MP3文件,为用户提供良好的体验,越来越引起人们的关注。Hadoop的出现,提供了一种新的思路。但由于Hadoop自身并不适合海量小文件处理,提出一种基于Hadoop的海量MP3文件存储架构,充分利用MP3文件自身包含的丰富描述信息,通过预处理模块的归类算法,将小文件归并到Sequence File中,同时引入高效的索引机制,很好地解决了小文件过多时NameNode内存瓶颈问题。实验证明,该架构能达到良好的效果。

关键词: Hadoop, MP3, 小文件, 富元数据, 存储架构

Abstract: MP3 as the de facto standard for digital music, the number of files is quite large and user access requirements rapidly grow up. How to effectively store and manage vast amounts of MP3 files to provide good user experience has become a matter of bigger concern. The emergence of Hadoop provides us with new ideas. However, because Hadoop itself is not suitable for handling massive small files, this paper presented a Hadoop-based storage architecture for massive MP3 files, fullly using the MP3 file’s rich metadata. The classification algorithm by pre-processing module would merged small files into sequence file, and the introduction of efficient indexing mechanism served as a good solution to the problem of small files. The experimental results show that the approach can achieve a better performance.

Key words: Hadoop, MP3, small files, rich metadata, storage architecture