create, help, list, status, activate, deactivate : GlusterFS Snapshots CLI Part 1

After discussing what GlusterFS snapshots are, what are their pre-requisites, and  what goes behind creation of a snapshot, it's time we actually created one, and familiarize ourselves with it.

To begin with let's create a volume called test_vol.

# gluster volume create test_vol replica 3 VM1:/brick/brick-dirs/brick VM2:/brick/brick-dirs/brick VM3:/brick/brick-dirs/brick
volume create: test_vol: success: please start the volume to access data
#
# gluster volume start test_vol
volume start: test_vol: success
#
# gluster volume info test_vol

Volume Name: test_vol
Type: Replicate
Volume ID: 09e773c9-e846-4568-a12d-6efb1cecf8cf
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: VM1:/brick/brick-dirs/brick
Brick2: VM2:/brick/brick-dirs/brick
Brick3: VM3:/brick/brick-dirs/brick
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
#

As you can see, we created a 1x3 replica volume, and started it. We are now primed to take our snapshot of this volume. But before, we do so let's add some data to the volume.

# mount -t glusterfs VM1:/test_vol /mnt/test-vol-mnt/
#
# cd /mnt/test-vol-mnt
#
# ls -lrt
total 0
# touch file1
# ls -lrt
total 0
-rw-r--r-- 1 root root 0 Jun 24 13:39 file1
#

So we have successfully mounted our volume and created(touched) a file called file1. Now we will take a snapshot of 'test_vol' and we will call it 'snap1'.

# gluster snapshot create snap1 test_vol
snapshot create: success: Snap snap1_GMT-2016.06.24-08.12.42 created successfully
#

That's weird isn't it. I asked it to create a snapshot called snap1, and it created a snapshot called snap1_GMT-2016.06.24-08.12.42. What happened is it actually created a snapshot called snap1 and appended the snap's name with the timestamp of it's creation. This is the default naming convention of GlusterFS snapshots, and like everything else it it so for a couple of reasons.

• This naming format is essential to support Volume Shadow Copy Service Support in GlusterFS volumes.
• The reason for keeping it as default naming convention is that it is more informative than just a name. Scrolling through a list of snapshots not only gives you the thoughtful name you have chosen for it, but also the time the snapshot was created, which makes it so much more relatable to you, and gives you more clarity to decide what to do with the said snapshot.

But if it still look's icky to you, as it does to a lot of people, you can choose to  not have the timestamp appended by adding the no-timestamp option in the create command.

# gluster snapshot create snap1 test_vol no-timestamp
snapshot create: success: Snap snap1 created successfully
#

So there you go. Congratulation on creating your first GlusterFS snapshot. Now what do you do with it, or rather what all can you do with it. Let's ask for some help.

# gluster snapshot help
snapshot activate <snapname> [force] - Activate snapshot volume.
snapshot clone <clonename> <snapname> - Snapshot Clone.
snapshot config [volname] ([snap-max-hard-limit <count>] [snap-max-soft-limit <percent>]) | ([auto-delete <enable|disable>])| ([activate-on-create <enable|disable>]) - Snapshot Config.
snapshot create <snapname> <volname> [no-timestamp] [description <description>] [force] - Snapshot Create.
snapshot deactivate <snapname> - Deactivate snapshot volume.
snapshot delete (all | snapname | volume <volname>) - Snapshot Delete.
snapshot help - display help for snapshot commands
snapshot info [(snapname | volume <volname>)] - Snapshot Info.
snapshot list [volname] - Snapshot List.
snapshot restore <snapname> - Snapshot Restore.
snapshot status [(snapname | volume <volname>)] - Snapshot Status.
#

Quite the buffet isn't it. So let's first see what snapshots do we have here. gluster snapshot list will do the trick for us.

# gluster snapshot list
snap1_GMT-2016.06.24-08.12.42
snap1
#
# gluster snapshot list test_vol
snap1_GMT-2016.06.24-08.12.42
snap1
#

The list command will display all the snapshots in the trusted pool. Adding a volume's name along with the list command will list all snapshots of that particular volume only. As we have only one volume now, it shows the same result for both. It helps provide more clarity when you have a couple of volumes, and each volume has a number of snapshots.

We have previously discussed that a GlusterFS snapshot is like a GlusterFS volume. Just like a regular volume you can mount it, delete it, and even see it's status. So let's see the status of our snapshots.

# gluster snapshot status

Snap Name : snap1_GMT-2016.06.24-08.12.42
Snap UUID : 26d1455d-1d58-4c39-9efa-822d9397088a

    Brick Path        :   VM1:/var/run/gluster/snaps/f4b2ae1fbf414c8383c3b198dd42e7d7/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   95.81
    LV Size           :   616.00m


    Brick Path        :   VM2:/var/run/gluster/snaps/f4b2ae1fbf414c8383c3b198dd42e7d7/brick2/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.45
    LV Size           :   616.00m


    Brick Path        :   VM3:/var/run/gluster/snaps/f4b2ae1fbf414c8383c3b198dd42e7d7/brick3/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.43
    LV Size           :   616.00m


Snap Name : snap1
Snap UUID : 73489d9b-c370-4687-8be9-fc094ee78d0a

    Brick Path        :   VM1:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   95.81
    LV Size           :   616.00m


    Brick Path        :   VM2:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick2/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.45
    LV Size           :   616.00m


    Brick Path        :   VM3:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick3/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.43
    LV Size           :   616.00m

As with the volume status command, the snapshot status command also shows the status of all the snapshot bricks of all snapshots. Adding the snapname in the status command displays the status of only that particular snapshot.

# gluster snapshot status snap1

Snap Name : snap1
Snap UUID : 73489d9b-c370-4687-8be9-fc094ee78d0a

    Brick Path        :   VM1:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   95.81
    LV Size           :   616.00m


    Brick Path        :   VM2:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick2/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.45
    LV Size           :   616.00m


    Brick Path        :   VM2:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick3/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.43
    LV Size           :   616.00m

Similar to the snapshot list command adding the volname instead of the snapname in the status command displays the status of all snapshots of that particular volume.

The status itself gives us a wealth of information about each snapshot brick like the volume group, the data percentage, the LV Size. It also tells us if the brick is running or not, and if it is what is the PID of the brick. Interestingly we see that none of the bricks are running. This is the default behaviour of GlusterFS snapshots, where a newly created snapshot is in deactivated state(analogous to the Created/Stopped state of a GlusterFS volume), where none of it's bricks are running. In order to start the snap brick process we will have to activate the snapshot.

# gluster snapshot activate snap1
Snapshot activate: snap1: Snap activated successfully
#
# gluster snapshot status snap1

Snap Name : snap1
Snap UUID : 73489d9b-c370-4687-8be9-fc094ee78d0a

    Brick Path        :   VM1:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   Yes
    Brick PID         :   29250
    Data Percentage   :   95.81
    LV Size           :   616.00m


    Brick Path        :   VM1:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick2/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   Yes
    Brick PID         :   12616
    Data Percentage   :   3.45
    LV Size           :   616.00m


    Brick Path        :   VM1:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick3/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   Yes
    Brick PID         :   3058
    Data Percentage   :   3.43
    LV Size           :   616.00m

After the snapshot is activated, we can see the the bricks are running and their respective PIDs. The snapshot can also be deactivated again by using the deactivate command.

# gluster snapshot deactivate snap1
Deactivating snap will make its data inaccessible. Do you want to continue? (y/n) y
Snapshot deactivate: snap1: Snap deactivated successfully
#
# gluster snapshot status snap1

Snap Name : snap1
Snap UUID : 73489d9b-c370-4687-8be9-fc094ee78d0a

    Brick Path        :   VM1:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   95.81
    LV Size           :   616.00m


    Brick Path        :   VM2:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick2/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.45
    LV Size           :   616.00m


    Brick Path        :   VM3:/var/run/gluster/snaps/d5171e51e1ef407292ee4e24677385cb/brick3/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   3.43
    LV Size           :   616.00m

Uptill now we have barely grazed the surface. There's delete, restore, config, and a whole lot more. We will be covering these in future posts.

GlusterFS Snapshots And Their Prerequisites

Long time, no see huh!!! This post has been pending on my part for a while now, partly because I was busy and partly because I am that lazy. But it's a fairly important post as it talks about snapshotting the GlusterFS volumes. So what are these snapshots and why are they so darn important. Let's find out...

Wikipedia says, 'a snapshot is the state of a system at a particular point in time'. In filesystems specifically, a snapshot is a 'backup' (a read only copy of the data set frozen at a point in time). Obviously, it's not a full backup of the entire dataset, but it's a backup nonetheless, which makes it pretty important. Now moving on to GlusterFS snapshots. GlusterFS snapshots, are point-in-time, read-only, crash consistent copies, of GlusterFS volumes. They are also online snapshots, and hence the volume and it's data continue to be available to the clients, while the snapshot is being taken.

GlusterFS snapshots are thinly provisioned LVM based snapshots, and hence they have certain pre-requisites. A quick look at the product documentation tells us what those pre-requisites. For a GlusterFS volume, to be able to support snapshots, it needs to meet the following pre-requisites:

• Each brick of the GlusterFS volume, should be on an independent, thinly-provisioned LVM.
• A brick's lvm should not contain any data other than the brick's.
• None of the bricks should be on a thick LVM
• gluster version should be 3.6 and above (duh!!)
• The volume should be started.
• All brick processes must be up and running.

Now that I have laid out the rules above, let me give you their origin story as well. As in, how do the GlusterFS snapshots internally enable you to take a crash-consistent backup using thinly-provisioned LVM in a space efficient manner. We start by having a look at a GlusterFS volume, whose bricks are on independent, thinly-provisioned LVMs.

In the above diagram, we can see that GlusterFS volume test_vol comprises of two bricks, Brick1 and Brick2. Both the bricks are mounted on independent, thinly-provisioned LVMs. When the volume is mounted, the client process maintains a connection to both the bricks. This is as much summary of GlusterFS volumes, as is needed for this post. A GlusterFS snapshot, is also internally a GlusterFS volume with the exception that, it is a read-only volume and it is treated differently than a regular volume in certain aspects.

When we take a snapshot (say snap1) of the GlusterFS volume test_vol, following set of things happen in the background:

•  It is checked if the volume is in started state, and if so are all the brick processes up and running.
• At this point in time, we barrier certain fops, in order to make the snapshots crash-consistent. What it means is even though it is an online snapshot, certain write fops will be barriered for the duration of the snapshot. The fops that are on the fly when the barrier is initiated will be allowed to complete, but the acknowledgement to the client will be pending till the snapshot creation is complete. The barriering has a default time-out window of 2 mins, within which if the snapshot is not complete, the fops are unbarriered, and we fail that particular snapshot.
• After successfully barriering fops on all brick processes, we proceed to take individual copy-on-write LVM snapshots of each brick. A copy-on-write snapshot LVM snapshot ensures a fast, space-efficient backup of the data currently on the brick. These LVM snapshots reside in the same LVM thinpool as the GLusterFS brick LVMs.
• Once this snapshot is taken, we carve bricks out of these LVM snapshots, and create a snapshot volume out of those bricks.
• Once the snapshot creation is complete, we unbarrier the GlusterFS volume.

As can be seen in the above diagram, the snapshot creation process has created a LVM snapshot for each LVM, and these snapshots lie in the same thinpool as the LVM. Then we carve bricks (Brick1" and Brick2") out of these snapshots, and create a snapshot volume called snap1.

This snapshot, snap1 is a read-only snapshot volume which can be:

• Restored to the original  volume test_vol.
• Mounted as a read-only volume and accessed.
• Cloned to create a writeable snapshot.
• Can be accessed via User-Servicable-Snapshots.

All these functionalities will be discussed in future posts, starting with the command line tools to create, delete and restore GlusterFS snapshots.

Volume Files and A Sneak Peak At Translators

In my last post, we went through the three vanilla types of volumes : Distribute, Replicate and Stripe, and now we have a basic understanding of what each of them does. I also mentioned we would be creating volumes, which are a mix of these three types. But before we do so, let's have a look at volume files.

Whenever a volume is created, subsequent volume files are also created. Ideally volume files are located in a directory(bearing the same name as the volume name) inside /var/lib/glusterd/vols/. Let's have a look at the volume files, for the distribute volume(test-vol) we created last time.

# gluster volume info

Volume Name: test-vol
Type: Distribute
Volume ID: 5d28ca28-9363-4b79-b922-5f28d0c0db65
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2

# cd /var/lib/glusterd/vols/
# ls -lrt
total 4
drwxr-xr-x. 4 root root 4096 Jan 31 14:20 test-vol
# cd test-vol/
# ls -lrt
total 32
-rw-------. 1 root root 1406 Jan 31 14:19 test-vol.Gotham.home-asengupt-node1.vol
-rw-------. 1 root root 1406 Jan 31 14:19 test-vol.Gotham.home-asengupt-node2.vol
-rw-------. 1 root root 1349 Jan 31 14:19 trusted-test-vol-fuse.vol
-rw-------. 1 root root 1121 Jan 31 14:20 test-vol-fuse.vol
drwxr-xr-x. 2 root root   80 Jan 31 14:20 run
-rw-------. 1 root root  305 Jan 31 14:20 info
drwxr-xr-x. 2 root root   74 Jan 31 14:20 bricks
-rw-------. 1 root root   12 Jan 31 14:20 rbstate
-rw-------. 1 root root   34 Jan 31 14:20 node_state.info
-rw-------. 1 root root   16 Jan 31 14:20 cksum
#

As I have the bricks in the same machine as the mount, hence we are seeing all the volume files here. The test-vol.Gotham.home-asengupt-node1.vol and test-vol.Gotham.home-asengupt-node2.vol are the volume files for Brick1 and Brick2 respectively. The volume file for test-vol volume is trusted-test-vol-fuse.vol. Let's have a look inside :

# cat trusted-test-vol-fuse.vol
volume test-vol-client-0
    type protocol/client
    option password 010f5d80-9d99-4b7c-a39e-1f964764213e
    option username 6969e53a-438a-4b92-a113-de5e5b7b5464
    option transport-type tcp
    option remote-subvolume /home/asengupt/node1
    option remote-host Gotham
end-volume

volume test-vol-client-1
    type protocol/client
    option password 010f5d80-9d99-4b7c-a39e-1f964764213e
    option username 6969e53a-438a-4b92-a113-de5e5b7b5464
    option transport-type tcp
    option remote-subvolume /home/asengupt/node2
    option remote-host Gotham
end-volume

volume test-vol-dht
    type cluster/distribute
    subvolumes test-vol-client-0 test-vol-client-1
end-volume

volume test-vol-write-behind
    type performance/write-behind
    subvolumes test-vol-dht
end-volume

volume test-vol-read-ahead
    type performance/read-ahead
    subvolumes test-vol-write-behind
end-volume

volume test-vol-io-cache
    type performance/io-cache
    subvolumes test-vol-read-ahead
end-volume

volume test-vol-quick-read
    type performance/quick-read
    subvolumes test-vol-io-cache
end-volume

volume test-vol-md-cache
    type performance/md-cache
    subvolumes test-vol-quick-read
end-volume

volume test-vol
    type debug/io-stats
    option count-fop-hits off
    option latency-measurement off
    subvolumes test-vol-md-cache
end-volume
#

This is what a volume file looks like. It's actually an inverted graph, of the path that the data from the mount point is supposed to follow. Savvy? No? Ok, Let's have a closer look at it. It is made up of a number of sections, each of which begin with "volume test-vol-xxxxx" and end with "end-volume". Each section stores information (type, options, subvolumes etc) for the respective translators(we will come back to what they are in a minute).

For example : Let's say a read fop(file-operation) was attempted at the mount point. The request information (type of fop:read, filename:the file the user tried to read etc.) will be passed on from one translator to another, starting from the io-stats translator at the bottom to either of the client translators at the top. Similarly the response will be transferred back from the client translator, all the way down to the io-stats and finally to the user.

So what are these translators. A translator is a module, which has a very specific purpose. It is to receive data, perform the necessary operations, and pass the data down to the next translator. That's about it in a nutshell. For example let's look at the dht-translator in the above volume file.

volume test-vol-dht
    type cluster/distribute
    subvolumes test-vol-client-0 test-vol-client-1
end-volume

"test-vol" is a distribute type volume, and hence has a dht translator. DHT a cluster translator, as is visible in it's "type". We know that in a distribute volume, a hashing algorithm, decides in which of the "subvolumes" is the data actually present. DHT translator is the one who does that for us.

The dht-translator will receive our read-fop, along with the filename. Based on the filename, the hashing algorithm, finds out the correct subvolume (among the two "test-vol-client-0" and "test-vol-client-1"), and passes the read-fop to the subsequent translator. That is how every other translator in the graph also works(recieve the data, perform it's part of the processing, and pass the data on to the next translator). As is quite visible here, the concept of translators provides us with a lot of modularity.

The volume files are created by the volume-create command, and based on the type of the volume and it's options, a graph is build with the appropriate translators. But we can edit an existing volume-file (add, remove, or modify a couple of translators), and the volume will change behaviour accordingly. Let's try that. Currently "test-vol" is a distribute volume. As per the behaviour, any file that is created, will be present in one of the bricks.

# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# cd /mnt/test-vol-mnt/
# ls -lrt
total 0
# touch file1
# ls -lrt
total 0
-rw-r--r--. 1 root root 0 Jan 31 15:57 file1
# ls -lrt /home/asengupt/node1/
total 0
# ls -lrt /home/asengupt/node2/
total 0
-rw-r--r--. 2 root root 0 Jan 31 15:57 file1
#

The dht-translator, created the file in node2 only. Let's edit the volume-file for test-vol(/var/lib/glusterd/vols/test-vol/trusted-test-vol-fuse.vol). But before that we need to stop the volume and unmount it.

# gluster volume stop test-vol
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test-vol: success
# umount /mnt/test-vol-mnt 

Then let's edit the volume file, and replace the dht-translator with a replicate translator.

# vi /var/lib/glusterd/vols/test-vol/trusted-test-vol-fuse.vol 
**********Necessary Changes in the volume file************
volume test-vol-afr
    type cluster/replicate
    subvolumes test-vol-client-0 test-vol-client-1
end-volume

volume test-vol-write-behind
    type performance/write-behind
    subvolumes test-vol-afr
end-volume
**********************************************************

Now start the volume, and mount it again.

# gluster volume start test-vol
volume start: test-vol: success
# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/

Let's create a file at this mount-point and check the behaviour of "test-vol".

# cd /mnt/test-vol-mnt/
# ls -lrt
total 0
-rw-r--r--. 1 root root 0 Jan 31 15:57 file1
# touch file2
# ls -lrt /home/asengupt/node1/
total 4
-rw-r--r--. 2 root root 0 Jan 31 16:06 file1
-rw-r--r--. 2 root root 0 Jan 31 16:06 file2
# ls -lrt /home/asengupt/node2/
total 4
-rw-r--r--. 2 root root 0 Jan 31 16:06 file1
-rw-r--r--. 2 root root 0 Jan 31 16:06 file2
#

Now we have, the same set of files in all the bricks, as is the behaviour of a replicate volume. We also observe that not only did it create the new file(file2) in all the bricks, but when we re-started the volume after changing the volume file, it also created a copy of the existing file(file1) in all the bricks. Let's check the volume info.

# gluster volume info 
  
Volume Name: test-vol 
Type: Distribute 
Volume ID: 5d28ca28-9363-4b79-b922-5f28d0c0db65 
Status: Started 
Number of Bricks: 2 
Transport-type: tcp 
Bricks: 
Brick1: Gotham:/home/asengupt/node1 
Brick2: Gotham:/home/asengupt/node2

It should be noted that, because of the changes we made in the volume file, though the behaviour of the volume changes, the volume-info still reflects the original details of the volume.

As promised we will create a mix of the vanilla volume-types, i.e a distributed-replicate volume.

# gluster volume create mix-vol replica 2 Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2 Gotham:/home/asengupt/node3 Gotham:/home/asengupt/node4
Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
Do you still want to continue creating the volume?  (y/n) y
volume create: mix-vol: success: please start the volume to access data
# gluster volume start mix-vol;
volume start: mix-vol: success
# gluster volume info

Volume Name: mix-vol
Type: Distributed-Replicate
Volume ID: 2fc6f11e-254e-444a-8179-43da62cc56e9
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2
Brick3: Gotham:/home/asengupt/node3
Brick4: Gotham:/home/asengupt/node4
#

As it is a distributed-replicate volume, the distribute translator should have two sub-volumes, each of which in turn is a replicate translator, having two bricks each as it's sub-volumes. A quick look at the volume file(/var/lib/glusterd/vols/mix-vol/trusted-mix-vol-fuse.vol) will give us more clarity. This is what the cluster translators in the volume file will look like :

volume mix-vol-replicate-0
    type cluster/replicate
    subvolumes mix-vol-client-0 mix-vol-client-1
end-volume

volume mix-vol-replicate-1
    type cluster/replicate
    subvolumes mix-vol-client-2 mix-vol-client-3
end-volume

volume mix-vol-dht
    type cluster/distribute
    subvolumes mix-vol-replicate-0 mix-vol-replicate-1
end-volume

As we can see the dht-translator has two replicate translators as its sub-volumes : "mix-vol-replicate-0" and "mix-vol-replicate-1". So every file created at the mount-point will be sent to either one of the replicate sub-volumes, by the dht-translator. Each replicate-subvolume has two bricks as it's own sub-volumes. After the write-fop is sent to the appropriate replicate-subvolume, the replicate translator will create a copy each, in both the bricks listed as it's sub-volumes. Let's check this behaviour :

# mount -t glusterfs Gotham:/mix-vol /mnt/test-vol-mnt/
# cd /mnt/test-vol-mnt/
# touch file1
# ls -lrt /home/asengupt/node1/
total 0
# ls -lrt /home/asengupt/node2/
total 0
# ls -lrt /home/asengupt/node3/
total 0
-rw-r--r--. 2 root root 0 Jan 31 16:31 file1
# ls -lrt /home/asengupt/node4/
total 0
-rw-r--r--. 2 root root 0 Jan 31 16:31 file1
#

Similarly, a distributed-stripe or a replicated-stripe volume can also be created.  Jeff Darcy's blog has an awesome set of articles on translators. It's  a great read :

•  Translator 101 Lesson 1: Setting the Stage
•  Translator 101 Lesson 2: init, fini, and private context
•  Translator 101 Lesson 3: This Time For Real
•  Translator 101 Lesson 4: Debugging a Translator
• ss

EDIT : The above links are broken, but the same information is provided in glusterfs/doc/developer-guide/translator-development.md in the source code. Thanks to Jo for pointing out the same.

Volumes

I feel it's safe to say, that we now have a fair idea of what GlusterFS is, and we are pretty comfortable installing GlusterFS, and creating a volume.
Let's create a volume with two local directories as two bricks.

# gluster volume create test-vol Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2
volume create: test-vol: success: please start the volume to access data
# gluster volume start test-vol;
volume start: test-vol: success

Let's mount this volume, and create a file in that volume.

# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# touch /mnt/test-vol-mnt/file1
# cd /mnt/test-vol-mnt/
# ls -lrt
total 1
-rw-r--r--. 1 root root   0 Jan 10 14:40 file1

Now where does this file really get created in the backend. Let's have a look at the two directories we used as bricks(subvolumes):

# cd /home/asengupt/node1
# ls -lrt
total 0
# cd ../node2/
# ls -lrt
total 1
-rw-r--r--. 1 root root   0 Jan 10 14:40 file1

So the file we created at the mount-point(/mnt/test-vol-mnt), got created in one of the bricks. But why in this particular brick, why not the other one? The answer to that question lies in the volume information.

# gluster volume info
 
Volume Name: test-vol
Type: Distribute
Volume ID: 5d28ca28-9363-4b79-b922-5f28d0c0db65
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2

It gives us a lot of info. While creating a volume we have the liberty of providing a number of options like the transport-type, the volume-type, etc. which eventually decides the behaviour of the volume. But at this moment what most interests us is the type. It says that our volume "test-vol" is a distributed volume. What does that mean?

The type of a volume decides, how exactly the volume stores the data in the bricks. A volume can be of the following types :

• Distribute : A distribute volume is one, in which all the data of the volume, is distributed throughout the bricks. Based on an algorithm, that takes into account the size available in each brick, the data will be stored in any one of the available bricks. As our "test-vol" volume is a distributed volume, so based on the algorithm "file1" was created in node2. The default volume type is distribute, hence test-vol is distribute.
  
• Replicate : In a replicate volume, the data is replicated(duplicated) over every brick, based on the brick number. The number of bricks must be a multiple of the replica count. So when "file1" is created in a replicate volume, having two bricks, it will be stored in brick1, and then replicated to brick2. So the file will be present in both the bricks. Let's create one and see for ourselves.
  

  # gluster volume create test-vol replica 2 Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2
Multiple bricks of a replicate volume are present on the same server. This setup is not optimal.
Do you still want to continue creating the volume?  (y/n) y
volume create: test-vol: success: please start the volume to access data
# gluster volume start test-vol
volume start: test-vol: success
# gluster volume info
 
Volume Name: test-vol
Type: Replicate
Volume ID: bfb685e9-d30d-484c-beaf-e5fd3b6e66c7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2
# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# touch /mnt/test-vol-mnt/file1
# cd /mnt/test-vol-mnt/
# ls -lrt
total 0
-rw-r--r--. 1 root root 0 Jan 10 14:58 file1
# ls -lrt /home/asengupt/node1/
total 0
-rw-r--r--. 2 root root 0 Jan 10 14:58 file1
# ls -lrt /home/asengupt/node2/
total 0
-rw-r--r--. 2 root root 0 Jan 10 14:58 file1

• Stripe : A stripe volume is one, in which the data being stored in the backend is striped into units of a particular size, among the bricks. The default unit size is 128KB, but it's configurable. If we create a striped volume of stripe count 3, and then create a 300 KB file at the mount point, the first 128KB will be stored in the first sub-volume(brick in our case), the next 128KB in the second, and the remaining 56KB in the third. The number of bricks should be a multiple of the stripe count.
  

  # gluster volume create test-vol stripe 3 Gotham:/home/asengupt/node1 Gotham:/home/asengupt/node2 Gotham:/home/asengupt/node3
volume create: test-vol: success: please start the volume to access data
# gluster volume start test-vol
volume start: test-vol: success
# gluster volume info
 
Volume Name: test-vol
Type: Stripe
Volume ID: c5aa1590-2f6e-464d-a783-cd9bc222db30
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: Gotham:/home/asengupt/node1
Brick2: Gotham:/home/asengupt/node2
Brick3: Gotham:/home/asengupt/node3
# mount -t glusterfs Gotham:/test-vol /mnt/test-vol-mnt/
# cd /mnt/test-vol-mnt/
# ls -lrt
total 0
# cp /home/asengupt/300KB_File .
# ls -lrt
total 308
-rwxr-xr-x. 1 root root 307200 Jan 11 12:46 300KB_File
# ls -lrt /home/asengupt/node1/
total 132
-rwxr-xr-x. 2 root root 131072 Jan 11 12:46 300KB_File
# ls -lrt /home/asengupt/node2/
total 132
-rwxr-xr-x. 2 root root 262144 Jan 11 12:46 300KB_File
# ls -lrt /home/asengupt/node3/
total 48
-rwxr-xr-x. 2 root root 307200 Jan 11 12:46 300KB_File

  Why do we see that the first sub-volume indeed has 128kb of data, but the second and third sub-volumes contain 256KB, and 300KB respectively?
  That's because of holes. It means that the filesystem just pretends that at a particular place in the file there is just zero bytes, but no actual disk sectors are reserved for that place in the file. To proof this let's check the disk usage.
  

  # cd /home/asengupt
# du | grep node.$
136    ./node2
136    ./node1
52    ./node3
0    ./node4

  Here we observe, that node1 and node2 indeed have 128KB of data, while node3 has 44KB. The additional 8KB present in these directories, are glusterfs system files.

Apart from these three vanilla types of volume, we can also create a volume which is a mix of these types. We will go through these and the respective volume files in the next post.

GlusterFS - The Product

I can only think of three reasons why you would be reading this :

• You are a developer, interested in contributing to GlusterFS : If this is where you belong, you are in the right place. View this space, not as a guide, but as a peer's journal, and familiarize yourself with GlusterFS as I too continue to do the same.
• You are an existing contributor to GlusterFS, and you are still reading this, wondering what am I rambling about : If you are one of the elites, please keep in mind that I too am in the process of understanding the product right now, and the purpose of this communication is only to pen down my understanding of the product as I continue to work on it. So while I am at it, I am sure there will be times when the information I put down here, might be inadequate. In such cases, I request you to kindly provide comments below, and help make this forum more valuable.
• I have become so incredibly famous that you googled me, and landed up here : In case that this is true, cheers to me :p, If not, I guess it's pretty safe to assume, that you belong to either of the above two groups.

Before one starts to work on and contribute to a product, what's most essential is a fundamental overview of the product. So in this post, we will go through a very brief introduction of GlusterFS, followed by a hands-on of pulling in the source code, building and installing the product, and finally creating a volume.

So what is GlusterFS? Well, when we Google GlusterFS, this is what we find..
Wikipedia says "GlusterFS is a scale-out NAS file system.". The gluster-community says "GlusterFS is a powerful network/cluster file-system written in user space which uses FUSE to hook itself with VFS layer. GlusterFS takes a layered approach to the file system, where features are added/removed as per the requirement. Though GlusterFS is a File System, it uses already tried and tested disk file systems like ext3, ext4, xfs, etc. to store the data. It can easily scale up to petabytes of storage which is available to user under a single mount point."

Kudos to you if you understood all of it. If not let's start from the start. In the simplest terms we can say GlusterFS is a file system. So when you hear file system, you start thinking ext3, ext4, xfs right? Well they are disk file systems, which basically manage every file-system related operation(fops) in their own approach, and each of them have evolved over the years, to become quite stable and mature. But GlusterFS is not a disk file system. Instead of re-inventing the wheel, GlusterFS uses any of these disk files systems(xfs is recommended), as the back-end file system to perform all file related operations, while it itself sits on top, and that too in userspace. So the obvious question that's haunting you now, is what does it do at the top, that these fellows at the back-end are not doing. Well the answer lies in the one line that Wikipedia gave us : scale-out

What does scale-out really mean? Traditional storage systems, are single storage controller systems, which are connected to ideally a rack of disk drives. The processing power here will be fixed, and cannot be scaled as the storage capacity increases. A scale-out storage system, can consist of multiple modules. In this architecture, each module in the system, has processing as well as storage capacity, which also means, as the system scales, so does it's processing power. GlusterFS is scalable to petabytes and beyond, and provides a clustered scale-out solution.

Every module in GlusterFS is treated as a volume. A volume can consist of several bricks. A brick is a node or storage file-system, assigned to a volume. Now that you have a picture of bricks and volumes(doesn't matter even if it's a vague picture), try creating a volume of your own. Please go through this link to understand the developer's work-flow. If you are not familiar to git, you can have a look at this.

Now that you have gone through the developer's work-flow, registered yourself at Gerrit, and added your machine's public key to Gerrit, let's pull in the source code :

git clone ssh://[username@]git.gluster.org/glusterfs.git ~/Source/

Now you have a directory called Source in your home directory, which contains the source code. Before we begin to build the source code, please ensure you have the following libraries installed :

yum install libtool autoconf automake flex bison openssl openssl-devel libibverbs-devel readline-devel libxml2-devel libacl-devel sqlite-devel python-devel userspace-rcu-devel dbench nfs-utils yajl attr psmisc

Once these libraries are installed, you can build GlusterFS.

# ./autogen.sh
# ./configure
# make
# make install

To verify which version of GlusterFS is installed

# gluster --version

Congratulations you have now successfully built GlusterFS from the source code and installed it. Let's create a volume. Please do make sure, you have a hostname assigned to your machine. This avoids any hassles from dynamic IPs. For the moment, you don't need actual servers to create a volume.
Create two directories node1 and node2 in your home directory to be used as bricks. The command to create a new volume is

gluster volume create NEW-VOLNAME [stripe COUNT] [replica COUNT] [transport <tcp | rdma>] NEW-BRICK ...

For example :

# gluster volume create test-vol Hostname:/~/node1 Hostname:/~/node2

Start the volume:

# gluster volume start test-vol

To display the volume information :

# gluster volume info