Setting Up GlusterFS on Ubuntu 22.04
GlusterFS is a scalable, distributed filesystem that ensures high availability and fault tolerance. This guide walks through configuring GlusterFS on three Ubuntu 22.04 servers, enabling a replicated filesystem across nodes. We also cover maintenance tasks, performance optimizations, and troubleshooting techniques for production-ready setups.
Overview
We will configure GlusterFS across three servers:
- web: 10.0.0.2
- web-node1: 10.0.0.5
- web-node2: 10.0.0.6
By replicating data across these nodes, GlusterFS ensures fault tolerance and consistent access to data, even if a node fails.
Prerequisites
- Three Ubuntu 22.04 servers with sudo privileges.
- Network connectivity between the servers.
- Update the
/etc/hostsfile on each server with the following entries:
10.0.0.2 web
10.0.0.5 web-node1
10.0.0.6 web-node2
Installation Steps
Step 1: Install GlusterFS on All Nodes
Run the following commands on each server:
sudo apt update
sudo apt install glusterfs-server -y
sudo systemctl start glusterd
sudo systemctl enable glusterd
Step 2: Configure the GlusterFS Trusted Pool
From the primary node (web), probe the other nodes:
sudo gluster peer probe web-node1
sudo gluster peer probe web-node2
Tip: Run gluster peer status on any node to verify that all peers are connected.
Step 3: Create Brick Directories
Create brick directories on all nodes to store GlusterFS data:
sudo mkdir -p /glusterfs/brick1/gv0
Step 4: Create and Start the Volume
On the primary node, create a replicated GlusterFS volume:
sudo gluster volume create gv0 replica 3 web:/glusterfs/brick1/gv0 web-node1:/glusterfs/brick1/gv0 web-node2:/glusterfs/brick1/gv0
sudo gluster volume start gv0
Step 5: Mount the GlusterFS Volume
Install the GlusterFS client and mount the volume on each node:
sudo apt-get install glusterfs-client -y
sudo mkdir -p /mnt/glusterfs
sudo mount -t glusterfs web:/gv0 /mnt/glusterfs
To auto-mount at boot, add the following line to /etc/fstab:
web:/gv0 /mnt/glusterfs glusterfs defaults,_netdev 0 0
Testing the Setup
Create a test file on one node to verify data replication:
sudo touch /mnt/glusterfs/testfile.txt
Check the other nodes -- the file should appear on all of them.
Adding a New Node
Step 1: Prepare the New Node
Install GlusterFS and add its IP to /etc/hosts on all servers:
10.0.0.7 web-node3
Step 2: Add the Node to the Trusted Pool
sudo gluster peer probe web-node3
sudo gluster peer status
Network Optimization
Enhance performance by tuning TCP settings:
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sudo sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'
sudo sysctl -w net.core.netdev_max_backlog=250000
GlusterFS Maintenance
Rebalancing Volumes
Rebalance the volume after adding or removing bricks:
sudo gluster volume rebalance gv0 start
Checking Volume Health
Monitor volume health and heal inconsistencies:
sudo gluster volume heal gv0 info
sudo gluster volume heal gv0
Troubleshooting
Peer Not Connected
If a peer shows as disconnected, verify:
- Network connectivity between nodes.
- Ports 24007-24009 are open.
- The
glusterdservice is running on all nodes.
sudo systemctl status glusterd
Conclusion
By following this guide, we have set up a distributed and replicated GlusterFS filesystem across multiple nodes. This configuration ensures high availability and fault tolerance, making it suitable for demanding environments. We recommend regularly monitoring performance, maintaining volume health, and following best practices to ensure long-term reliability.