Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal

Dipayan DEV,Ripon PATGIRI

Front. Inform. Technol. Electron. Eng. 2016, 17 (1): 15-31. DOI: 10.1631/FITEE.1500015

Abstract

HTML

PDF (1278KB)

In this Exa byte scale era, data increases at an exponential rate. This is in turn generating a massive amount of metadata in the file system. Hadoop is the most widely used framework to deal with big data. Due to this growth of huge amount of metadata, however, the efficiency of Hadoop is questioned numerous times by many researchers. Therefore, it is essential to create an efficient and scalable metadata management for Hadoop. Hash-based mapping and subtree partitioning are suitable in distributed metadata management schemes. Subtree partitioning does not uniformly distribute workload among the metadata servers, and metadata needs to be migrated to keep the load roughly balanced. Hash-based mapping suffers from a constraint on the locality of metadata, though it uniformly distributes the load among NameNodes, which are the metadata servers of Hadoop. In this paper, we present a circular metadata management mechanism named dynamic circular metadata splitting (DCMS). DCMS preserves metadata locality using consistent hashing and locality-preserving hashing, keeps replicated metadata for excellent reliability, and dynamically distributes metadata among the NameNodes to keep load balancing. NameNode is a centralized heart of the Hadoop. Keeping the directory tree of all files, failure of which causes the single point of failure (SPOF). DCMS removes Hadoop’s SPOF and provides an efficient and scalable metadata management. The new framework is named ‘Dr. Hadoop’ after the name of the authors.

In Fig. 10a, the experiment is carried out on a Yahoo data trace, where Hadoop shows a steady state throughput initially. We kill the primary NameNode's daemon at t=100 s and eventually the whole HDFS goes offline. At around t=130 s, the NameNode is again restarted and it recovers itself from the check-pointed state from the secondary NameNode and repeats the log operations that it failed to perform. During the recovery phase, there are few spikes because the NameNode buffers all the requests until it recovers and batches them all together.

An experiment with the Microsoft data trace is shown in Fig. 10b. It shows a similar trend to the Yahoo trace for Hadoop. During the test for Dr. Hadoop, we kill two sets of three consecutive NameNodes, NameNode_1 to NameNode_3 and NameNode_6 to NameNode_8, at t=90 s to acquire the worst case scenario (with a probability of almost $10^{-8}$). This made a portion of the file system unavailable, and throughput of Dr. Hadoop gives a poorest value. At t=110 s we again start NameNode_1 to NameNode_3 to increase its throughput. Again, at t=130 s we start NameNode_6 to NameNode_8, which eventually re-stabilizes the situation.