DRBD

DRBD (Distributed Replicated Block Device) is a virtual block device that is distributed across two or more machines.

Installation

On a clean installation of RHEL 7 or equivalent, run updates, and ensure that you have NTP and a recent kernel installed.

From ElRepo

Get the DRBD packages from ELRepo:

# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
# yum install drbd84-utils kmod-drbd84

You should be able to load the drbd module:

# modprobe drbd
# lsmod

From Source

Install prerequisites:

# yum install gcc flex make autoconf m4

Get the source from their git repository. Releases from tarballs may fail depending on which kernel version you're using.

# git clone git://git.drbd.org/drbd-8.4.git
# cd drbd-8.4
# make
# make install

The make install should have loaded the module and you can verify this:

# lsmod

If it isn't loaded, load it using modprobe

# modprobe drbd

Compile the userspace utilities:

# git clone git://git.drbd.org/drbd-utils.git
# cd drbd-utils
# ./aclocal.sh
# ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
# make
# make install

Note: I got errors about man pages not being found. Ugh. Use the RPMs instead (it worked after a clean install...)

Configuration

This section will configure DRBD for use in a redundant MySQL setup.

DRBD syncs volumes. If you did not configure a partition or have a separate disk to mirror, you can create a file volume instead.

On all nodes:

# mkdir /var/drbd
# cd /var/drbd
 
## Create a 50GB volume
# dd if=/dev/zero of=data_volume.dat bs=1024 count=52428800

## Create a partition
# echo -e "n\n\n\n\nw\nq\n"

On all nodes, configure the DRBD config in /etc/drbd.d/*conf. You may want to disable the survey.

On all nodes, configure DRBD by creating a file in /etc/drbd.d/mysql.res with:

resource mysql {
	disk {
		on-io-error detach;
	}

	device    /dev/drbd0;
	disk      /dev/mapper/loop0p1;
	meta-disk internal;

	on $hostname {
		address 172.20.10.10:7700;
	}

	on $hostname {
		address 172.20.10.9:7700;
	}
}

TODO: Can you just use /dev/loop0 instead?!

Be sure to rename the hostnames so that it matches the actual machine's hostname. Otherwise, when you try creating the resource, you will get an error like:

'mysql' not defined in your config (for this host).

On all nodes, create the resource:

# drbdadm -v create-md mysql

If it complains about finding a filesystem, zero out the first few blocks using dd and try again or ignore and continue.

On all nodes, start the resource:

# drbdadm up mysql

You can check the status of the drbd volume:

# drbd-overview
0:mysql/0  Connected Secondary/Secondary Inconsistent/Inconsistent

The two volumes will not sync until one of them is set to be the primary.

On one of the nodes, force it to be the primary node.

# drbdadm -- --force primary mysql

On the primary node, create a file system:

# mkfs -t xfs /dev/drbd0

You can now mount the device on the primary node:

# mount /dev/drbd0 /mnt/drbd

At this stage, you should be seeing the two volumes beginning to sync:

# drbd-overview
 0:mysql/0  SyncSource Primary/Secondary UpToDate/Inconsistent /mnt/drbd xfs 35G 34M 35G 1% 
	[=>..................] sync'ed: 14.4% (30480/35580)M

Setting up MySQL

Install MySQL (or MariaDB)

# yum -y install mariadb mariadb-server

If you mounted the drbd volume to /mnt/drbd, you could make a symlink for /var/lib/mysql.

# ln -s /mnt/drbd/mysql /var/lib/mysql

Initialize the database:

# mysql_install_db

Corosync / CRM

You'll need crmsh if you want to follow some other guides. Install it using the repository at:

• http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/

# cd
# wget http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/python-pssh-2.3.1-4.2.x86_64.rpm http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/pssh-2.3.1-4.2.x86_64.rpm http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/crmsh-2.1.2-1.1.x86_64.rpm
# yum install *.rpm

Configure the cluster so we have a an IP address set, MySQL running, and the DRBD volume mounted.

# crm configure
# primitive p_ip_mysql ocf:heartbeat:IPaddr2 \
   params ip="172.20.10.8" cidr_netmask="24" \
   op monitor interval="30s"
# primitive p_drbd_mysql ocf:linbit:drbd \
   params drbd_resource="mysql" \
   op start timeout="90s" \
   op stop timeout="180s" \
   op promote timeout="180s" \
   op demote timeout="180s" \
   op monitor interval="30s" role="Slave" \
   op monitor interval="29s" role="Master"
# primitive p_fs_mysql ocf:heartbeat:Filesystem \
   params device="/dev/drbd1" \
     directory="/mnt/drbd" \
     fstype="xfs" \
     options="relatime" \
   op start timeout="60s" \
   op stop timeout="180s" \
   op monitor interval="60s" timeout="60s"
# primitive p_mysql ocf:heartbeat:mysql \
   params additional_parameters="--bind-address=172.20.10.8" \
     config="/etc/my.cnf" \
     pid="/var/run/mariadb/mariadb.pid" \
     socket="/var/lib/mysql/mysql.sock" \
     log="/var/log/mariadb/mariadb.log" \
   op monitor interval="20s" timeout="10s" \
   op start timeout="120s" \
   op stop timeout="120s"
# group g_mysql p_ip_mysql p_fs_mysql p_mysql
# ms ms_drbd_mysql p_drbd_mysql \
   meta notify="true" clone-max="2"
# colocation c_mysql_on_drbd inf: g_mysql ms_drbd_mysql:Master
# order o_drbd_before_mysql inf: ms_drbd_mysql:promote g_mysql:start
# commit

We'll need to configure STONITH.

You can see all the available STONITH devices using:

# stonith_admin -I

You can use any of the listed devices as part of your configuration. (Added after stonith:_________)

# crm configure
# property stonith-enabled=true
# property stonith-action=poweroff
# rsc_defaults resource-stickiness=100
# property no-quorum-policy=ignore
  
# primitive vm-fence-c1 stonith:fence_vmware_soap \
   params ipaddr="172.17.12.8" login="root" passwd="xxx" \
     port="OpenStack C1" ssl="1" retry_on="20" shell_timeout="10" \
     login_timeout="10" action="off" verbose="true"
  
# primitive vm-fence-c2 stonith:fence_vmware_soap \
   params ipaddr="172.20.1.27" login="root" passwd="xxx" \
     port="OpenStack C2" ssl="1" retry_on="20" shell_timeout="10" \
     login_timeout="10" action="off" verbose="true"
 
# location l-vm-c1 vm-fence-c1 -inf: os-c1
# location l-vm-c2 vm-fence-c2 -inf: os-c2
# commit

Other Commands

Change a node to a secondary on a specific resource (eg: mysql):

# drbdadm secondary mysql

To show the current role of a node on a specific resource:

# drbdadm role mysql

Split Brain

The two nodes will either be in the WFConnection or StandAlone state:

[root@os-c1 mnt]# drbd-overview 
0:mysql/0  StandAlone Secondary/Unknown UpToDate/DUnknown 

[root@os-c2 mnt]# drbd-overview 
0:mysql/0  StandAlone Primary/Unknown UpToDate/DUnknown

Make both nodes in the StandAlone state by disconnecting the resource if it is in the WFConnection state.

Set the two nodes to secondary

# drbdadm secondary mysql

The node which that is to have its data discarded will then need connect with the --discard-my-data option:

# drbdadm connect --discard-my-data mysql

The node that is to have its data preserved should just connect:

# drbdadm connect mysql

Both nodes should not be in the StandAlone state and should at this point be connected and syncing.

See Also

• https://drbd.linbit.com/users-guide/s-build-from-source.html
• http://unix-linux-server.blogspot.ca/2014/10/drbd-on-centos-70-oracle-linux-70-and.html
• http://wiki.centos.org/HowTos/Ha-Drbd (CentOS 5)
• http://wiki.openvz.org/HA_cluster_with_DRBD_and_Heartbeat