Volumes

I feel it's safe to say, that we now have a fair idea of what GlusterFS is, and we are pretty comfortable installing GlusterFS, and creating a volume.
Let's create a volume with two local directories as two bricks.

# gluster volume create test-vol Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2
volume create: test-vol: success: please start the volume to access data
# gluster volume start test-vol;
volume start: test-vol: success

Let's mount this volume, and create a file in that volume.

# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# touch /mnt/test-vol-mnt/file1
# cd /mnt/test-vol-mnt/
# ls -lrt
total 1
-rw-r--r--. 1 root root   0 Jan 10 14:40 file1

Now where does this file really get created in the backend. Let's have a look at the two directories we used as bricks(subvolumes):

# cd /home/asengupt/node1
# ls -lrt
total 0
# cd ../node2/
# ls -lrt
total 1
-rw-r--r--. 1 root root   0 Jan 10 14:40 file1

So the file we created at the mount-point(/mnt/test-vol-mnt), got created in one of the bricks. But why in this particular brick, why not the other one? The answer to that question lies in the volume information.

# gluster volume info
 
Volume Name: test-vol
Type: Distribute
Volume ID: 5d28ca28-9363-4b79-b922-5f28d0c0db65
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2

It gives us a lot of info. While creating a volume we have the liberty of providing a number of options like the transport-type, the volume-type, etc. which eventually decides the behaviour of the volume. But at this moment what most interests us is the type. It says that our volume "test-vol" is a distributed volume. What does that mean?

The type of a volume decides, how exactly the volume stores the data in the bricks. A volume can be of the following types :

• Distribute : A distribute volume is one, in which all the data of the volume, is distributed throughout the bricks. Based on an algorithm, that takes into account the size available in each brick, the data will be stored in any one of the available bricks. As our "test-vol" volume is a distributed volume, so based on the algorithm "file1" was created in node2. The default volume type is distribute, hence test-vol is distribute.
  
• Replicate : In a replicate volume, the data is replicated(duplicated) over every brick, based on the brick number. The number of bricks must be a multiple of the replica count. So when "file1" is created in a replicate volume, having two bricks, it will be stored in brick1, and then replicated to brick2. So the file will be present in both the bricks. Let's create one and see for ourselves.
  

  # gluster volume create test-vol replica 2 Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2
Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
Do you still want to continue creating the volume?  (y/n) y
volume create: test-vol: success: please start the volume to access data
# gluster volume start test-vol
volume start: test-vol: success
# gluster volume info
 
Volume Name: test-vol
Type: Replicate
Volume ID: bfb685e9-d30d-484c-beaf-e5fd3b6e66c7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2
# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# touch /mnt/test-vol-mnt/file1
# cd /mnt/test-vol-mnt/
# ls -lrt
total 0
-rw-r--r--. 1 root root 0 Jan 10 14:58 file1
# ls -lrt /home/asengupt/node1/
total 0
-rw-r--r--. 2 root root 0 Jan 10 14:58 file1
# ls -lrt /home/asengupt/node2/
total 0
-rw-r--r--. 2 root root 0 Jan 10 14:58 file1

• Stripe : A stripe volume is one, in which the data being stored in the backend is striped into units of a particular size, among the bricks. The default unit size is 128KB, but it's configurable. If we create a striped volume of stripe count 3, and then create a 300 KB file at the mount point, the first 128KB will be stored in the first sub-volume(brick in our case), the next 128KB in the second, and the remaining 56KB in the third. The number of bricks should be a multiple of the stripe count.
  

  # gluster volume create test-vol stripe 3 Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2 Gotham:/home/asengupt/node3
volume create: test-vol: success: please start the volume to access data
# gluster volume start test-vol
volume start: test-vol: success
# gluster volume info
 
Volume Name: test-vol
Type: Stripe
Volume ID: c5aa1590-2f6e-464d-a783-cd9bc222db30
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2
Brick3: Gotham:/home/asengupt/node3
# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# cd /mnt/test-vol-mnt/
# ls -lrt
total 0
# cp /home/asengupt/300KB_File .
# ls -lrt
total 308
-rwxr-xr-x. 1 root root 307200 Jan 11 12:46 300KB_File
# ls -lrt /home/asengupt/node1/
total 132
-rwxr-xr-x. 2 root root 131072 Jan 11 12:46 300KB_File
# ls -lrt /home/asengupt/node2/
total 132
-rwxr-xr-x. 2 root root 262144 Jan 11 12:46 300KB_File
# ls -lrt /home/asengupt/node3/
total 48
-rwxr-xr-x. 2 root root 307200 Jan 11 12:46 300KB_File

  Why do we see that the first sub-volume indeed has 128kb of data, but the second and third sub-volumes contain 256KB, and 300KB respectively?
  That's because of holes. It means that the filesystem just pretends that at a particular place in the file there is just zero bytes, but no actual disk sectors are reserved for that place in the file. To proof this let's check the disk usage.
  

  # cd /home/asengupt
# du | grep node.$
136    ./node2
136    ./node1
52    ./node3
0    ./node4

  Here we observe, that node1 and node2 indeed have 128KB of data, while node3 has 44KB. The additional 8KB present in these directories, are glusterfs system files.

Apart from these three vanilla types of volume, we can also create a volume which is a mix of these types. We will go through these and the respective volume files in the next post.