Please wait a minute...
Front. Inform. Technol. Electron. Eng.  2010, Vol. 11 Issue (5): 315-327    DOI: 10.1631/jzus.C0910445
    
Scalable high performance de-duplication backup via hash join
Tian-ming Yang, Dan Feng*, Zhong-ying Niu, Ya-ping Wan
Wuhan National Laboratory for Optoelectronics, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, China
Download:   PDF(242KB)
Export: BibTeX | EndNote (RIS)      

Abstract  Apart from high space efficiency, other demanding requirements for enterprise de-duplication backup are high performance, high scalability, and availability for large-scale distributed environments. The main challenge is reducing the significant disk input/output (I/O) overhead as a result of constantly accessing the disk to identify duplicate chunks. Existing inline de-duplication approaches mainly rely on duplicate locality to avoid disk bottleneck, thus suffering from degradation under poor duplicate locality workload. This paper presents Chunkfarm, a post-processing de-duplication backup system designed to improve capacity, throughput, and scalability for de-duplication. Chunkfarm performs de-duplication backup using the hash join algorithm, which turns the notoriously random and small disk I/Os of fingerprint lookups and updates into large sequential disk I/Os, hence achieving high write throughput not influenced by workload locality. More importantly, by decentralizing fingerprint lookup and update, Chunkfarm supports a cluster of servers to perform de-duplication backup in parallel; it hence is conducive to distributed implementation and thus applicable to large-scale and distributed storage systems.

Key wordsBackup system      De-duplication      Post-processing      Fingerprint lookup      Scalability     
Received: 22 July 2009      Published: 28 April 2010
CLC:  TP309.3  
Fund:  Project supported by the National Basic Research Program (973) of China  (No.  2004CB318201),  the  National  High-Tech  Research  and
Development Program (863) of China (No. 2008AA01A402), and the National  Natural  Science  Foundation  of  China  (Nos.  60703046  and
60873028)
Cite this article:

Tian-ming Yang, Dan Feng, Zhong-ying Niu, Ya-ping Wan. Scalable high performance de-duplication backup via hash join. Front. Inform. Technol. Electron. Eng., 2010, 11(5): 315-327.

URL:

http://www.zjujournals.com/xueshu/fitee/10.1631/jzus.C0910445     OR     http://www.zjujournals.com/xueshu/fitee/Y2010/V11/I5/315


Scalable high performance de-duplication backup via hash join

Apart from high space efficiency, other demanding requirements for enterprise de-duplication backup are high performance, high scalability, and availability for large-scale distributed environments. The main challenge is reducing the significant disk input/output (I/O) overhead as a result of constantly accessing the disk to identify duplicate chunks. Existing inline de-duplication approaches mainly rely on duplicate locality to avoid disk bottleneck, thus suffering from degradation under poor duplicate locality workload. This paper presents Chunkfarm, a post-processing de-duplication backup system designed to improve capacity, throughput, and scalability for de-duplication. Chunkfarm performs de-duplication backup using the hash join algorithm, which turns the notoriously random and small disk I/Os of fingerprint lookups and updates into large sequential disk I/Os, hence achieving high write throughput not influenced by workload locality. More importantly, by decentralizing fingerprint lookup and update, Chunkfarm supports a cluster of servers to perform de-duplication backup in parallel; it hence is conducive to distributed implementation and thus applicable to large-scale and distributed storage systems.

关键词: Backup system,  De-duplication,  Post-processing,  Fingerprint lookup,  Scalability 
[1] Aisha SIDDIQA , Ahmad KARIM , Abdullah GANI. Big data storage technologies: a survey[J]. Front. Inform. Technol. Electron. Eng., 2017, 18(8): 1040-1070.
[2] Zhi-bo Wang, Zhi Wang, Hong-long Chen, Jian-feng Li, Hong-bin Li, Jie Shen. HierTrack: an energy-efficient cluster-based target tracking system for wireless sensor networks[J]. Front. Inform. Technol. Electron. Eng., 2013, 14(6): 395-406.