Distributed file system between multiple servers is a thing I have planned for a long time, but I never got around to it because I first had to find the right filesystem for it.
After a lot of research, I found that GlusterFS was the right file system for me.
My plan was to use a distributed filesystem to share the content of my webservers to make sure all my webservers had the same content on their pages at all time and using some sort of high availability to make sure the content was always there.
To this I used a tool in Linux called Lsyncd before I set up my GlusterFS cluster, this worked well and did live syncing of all the servers using rsync. But there was one problem with it.
If I uploaded a lot of files (20+) to one webserver, and it started to sync to the other webserver before I was finished copying, the result would be that some of the files ended up corrupt, and this was a problem for me!

My distributed file system overview

Here is the overview of the setup I am making, it’s not pretty but I’m sure it’s a lot more easy to explain the setup using a simple mspaint drawing than with text!

As you can see in the picture above, I have build this with high availability in mind, since I want my websites to always be up and running!
Continue reading →