Providing Elastic storage to Hadoop

Task 7.1 : Integrating LVM with Apache Hadoop

Let’s get our hands dirty… and directly jump to the practical demo…

I assume that you are familiar with
✅ What is Apache Hadoop?
✅ What is the use of Apache Hadoop ?

If not, check this article it will give you an idea about it..

And to know about LVM check out World record holders Youtube channel

Task Description :

📌 Integrating LVM with Hadoop and providing Elasticity to DataNode Storage
📌 Increase or Decrease the Size of Static Partition in Linux.
📌 Automating LVM using python Script

Let’s proceed…

Required initial setup
- Master and Slave node configured

1️⃣ Attach 2 external Hard disk :

I am attaching 1 hard disk of 4 GB and another one of 8 GB
(below are the steps to attach 4 GB storage repeat same steps for 8 GB also)

Adding 4 GB Storage to Slave node

Use fdisk -l to check the available hard disk

We have 50 GB(main), 8 GB (external) and 4 GB (external) disk

2️⃣ Creating physical volume :

Command : pvcreate <disk_path>

3️⃣ Creating Volume Group :

Now we want only 10 GB space but non of our disk is of the same so we will merge both of them to form one volume group

Command : vgcreate <volume_group_name> <disk_path> <disk2_path>

Task7vg(newly created VG) have 12 GB of storage capacity in totality but we want only 10 GB

VG showing 12 GB storage

4️⃣ Creating Logical volume :

We need only 10 out of 12 GB storage, so we will create a logical volume with the same capacity

Command : lvcreate --size <size_in_GB> --name <lv_name> <vg_name>

Logical volume with 10GB capacity created

10 GB logical volume created

5️⃣ Making logical volume usable :

Although the volume is created, it can’t be used to store anything, to make it usable we need to do the following-

✔ Create partition
✔ Format them
✔ Mount them to desired directory

Before we proceed.. let’s check how much storage is contributed by slave

Slave contributes 50 GB

◼ To make partition -

◼ To format -

◼ To mount it to our desired folder (/datanode1)

Let’s verify slaves’s storage changed or not !!

Now slave contributes only 10 GB as specified

oh ! Yes….

Let’s have some more fun…

⭕ Increase storage capacity by 1 GB

Tring to increase storage capacity by 1GB on the fly, without shutting down anything

Command : lvextend --size +1G <lv_path>

Command : resize2fs /dev/iiecvg/mylv1

Let’s verify slave’s storage again…

Storage capacity increased by 1 GB

⭕ Decrease storage capacity to 6 GB

When we decrease the storage capacity there is a chance of data loss, so we need to

✔ Stop hadoop service of the slave

✔ Unmount the logical volume
✔ Check its health
✔ Resize the LV

✔ Mount it back
✔ Start hadoop service

Let’s verify slave’s storage again…

Slave storage capacity changed to 5 GB only

🔴 Cherry on the cake 🍒 Automation:

If you think this entire process is tedious use this script to automate 😎 the management of logical voume.

To use this you just need the basic understanding of LVM which you already got from this article itself.

This concludes our practical for providing elastic storage to the Apache Hadoop..

😎🤩 This blog is incomplete without Thanking my source of learning… I am greatly Thankful to Mr. Vimal Daga Sir and LinuxWorld Informatics Pvt. Ltd., under whose mentorship i am able to learn these most demanded technologies with such an ease.

🙂 Hopefully, this helped you for any doubts or suggestions, you can ping me on Linkedin I’ll be happy to help.

Keep Learning !! Keep sharing !!