Limit the storage of data node in Hadoop cluster
We know, the data node shares its storage in the Hadoop cluster; but there is no specific limit to the storage. The directory or the folder which is shared with the cluster can take up the whole size of hard disk in which the directory is created.
So, to solve this problem, we can use the concept of partition of hard disk.
I have attached a new hard disk of 8 GiB to the data node and have created two partitions of 4 GiB and 3.8 GiB; and used the 4 Gib partition to contribute storage to the hadoop cluster.
This is a step by step guide to learn how to limit the storage of data node in Hadoop cluster.
Let’s start
- First, we will create a partition in hard disk.
Keep the partition number and first sector default by just pressing the enter key.
2. Allocate the size of partition
3. Save the new partition that we have created by pressing w key
4. To check that the partition is created, use the fdisk -l command
Note: I have created one more partition(/dev/sdb2) so that I can use the remaining space for other purpose as well.
5. Now, we have to format the partition
6. Create a directory so that we can mount the partition and then mount the partition on it.
7. Now configure the hdfs-site.xml file and set value as name of the directory to which the partition is mounted.
Now, it’s done!
Let’s check
Start the Name node and Data nodes of the Hadoop cluster and use the hadoop dfsadmin -report command to check the storage contributed by data node.
We can see that data node with IP address xx.xx.xx.204 is contributing 3.87 GB storage.
We have our hard disk of 8 GB, but using the above method, we have restricted the contribution of storage to 4 GB.
Thank you for reading!
