Please wait a minute...
IMAGE/TABLE DETAILS
Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal
Dipayan DEV,Ripon PATGIRI
Front. Inform. Technol. Electron. Eng.    2016, 17 (1): 15-31.   DOI: 10.1631/FITEE.1500015
Abstract   HTML   PDF (1278KB)

In this Exa byte scale era, data increases at an exponential rate. This is in turn generating a massive amount of metadata in the file system. Hadoop is the most widely used framework to deal with big data. Due to this growth of huge amount of metadata, however, the efficiency of Hadoop is questioned numerous times by many researchers. Therefore, it is essential to create an efficient and scalable metadata management for Hadoop. Hash-based mapping and subtree partitioning are suitable in distributed metadata management schemes. Subtree partitioning does not uniformly distribute workload among the metadata servers, and metadata needs to be migrated to keep the load roughly balanced. Hash-based mapping suffers from a constraint on the locality of metadata, though it uniformly distributes the load among NameNodes, which are the metadata servers of Hadoop. In this paper, we present a circular metadata management mechanism named dynamic circular metadata splitting (DCMS). DCMS preserves metadata locality using consistent hashing and locality-preserving hashing, keeps replicated metadata for excellent reliability, and dynamically distributes metadata among the NameNodes to keep load balancing. NameNode is a centralized heart of the Hadoop. Keeping the directory tree of all files, failure of which causes the single point of failure (SPOF). DCMS removes Hadoop’s SPOF and provides an efficient and scalable metadata management. The new framework is named ‘Dr. Hadoop’ after the name of the authors.




View image in article
Fig. 7 Locality comparisons of paths at three levels over two traces in the cluster with 10 NameNodes: (a) Yahoo trace; (b) Microsoft Windows trace
Extracts from the Article
DCMS uses LpH which is the finest feature of Dr. Hadoop. Excellent namespace locality of Dr. Hadoop's MDS cluster is necessary to reduce the I/O request of a metadata lookup. In this experiment, we have used a parameter, locality =$\sum\nolimits_{j = 1}^m {{p_{ij}}} $, where $p_{ij}$ (either 0 or 1) represents whether a subtree (directory) path $p_{i}$ is stored in NameNode j or not. Basically, this metric shows the total number of times a subtree path is split across the NameNodes. Fig. 7 portraits the average namespace locality comparison of paths at three different levels on two given traces using three metadata distribution techniques.
Figs. 7a and 7b show that the performance of DCMS is significantly improved over FileHash and DirHash for the given two traces. This is because DCMS achieves optimal namespace locality using LpH; i.e., keys are assigned based on the order of the pathnames. In contrast, in the cases of FileHash and DirHash, the orders of pathnames are not maintained, so namespace locality is not preserved at all.
Other Images/Table from this Article