DRBD

DRBD (Distributed Replicated Block Device) is a virtual block device that is distributed across two or more machines.

Installation

On a clean installation of RHEL 7 or equivalent, run updates, and ensure that you have NTP and a recent kernel installed.

From ElRepo

Get the DRBD packages from ELRepo:

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum install drbd84-utils kmod-drbd84

You should be able to load the drbd module:

# modprobe drbd
# lsmod|grep drbd
drbd                  392583  0 
libcrc32c              12644  1 drbd

From Source

Install prerequisites:

yum install gcc flex make autoconf m4

Get the source from their git repository. Releases from tarballs may fail depending on which kernel version you're using.

git clone git://git.drbd.org/drbd-8.4.git
cd drbd-8.4
make
make install

The make install should have loaded the module and you can verify this:

lsmod | grep drbd

If it isn't loaded, load it using modprobe

modprobe drbd

Compile the userspace utilities:

git clone git://git.drbd.org/drbd-utils.git
cd drbd-utils
./aclocal.sh
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
make
make install

Note: I got errors about man pages not being found. Ugh. Use the RPMs instead (it worked after a clean install...)

Configuration

This section will configure DRBD for use in a redundant MySQL setup.

DRBD syncs volumes. If you did not configure a partition or have a separate disk to mirror, you can create a file volume instead.

On all nodes:

mkdir /var/drbd
cd /var/drbd

# Create a 50GB volume
dd if=/dev/zero of=data_volume.dat bs=1024 count=52428800

# Create a partition
echo -e "n\n\n\n\nw\nq\n" | fdisk data_volume.dat

# Create a device
losetup /dev/loop0 /var/drbd/data_volume.dat
kpartx -v -a /var/drbd/data_volume.dat
ls /dev/mapper/

mkdir -p /mnt/drbd
echo "/dev/drbd0 /mnt/drbd ext4 noauto 0 0 " >>  /etc/fstab

On all nodes, configure the DRBD config in /etc/drbd.d/*conf. You may want to disable the survey.

On all nodes, configure DRBD by creating a file in /etc/drbd.d/mysql.res with:

resource mysql {
	disk {
		on-io-error detach;
	}

	device    /dev/drbd0;
	disk      /dev/mapper/loop0p1;
	meta-disk internal;

	on $hostname {
		address 172.20.10.10:7700;
	}

	on $hostname {
		address 172.20.10.9:7700;
	}
}

TODO: Can you just use /dev/loop0 instead?!

Be sure to rename the hostnames so that it matches the actual machine's hostname. Otherwise, when you try creating the resource, you will get an error like:

'mysql' not defined in your config (for this host).

On all nodes, create the resource:

drbdadm -v create-md mysql

If it complains about finding a filesystem, zero out the first few blocks using dd and try again or ignore and continue.

On all nodes, start the resource:

drbdadm up mysql

You can check the status of the drbd volume:

drbd-overview
 0:mysql/0  Connected Secondary/Secondary Inconsistent/Inconsistent

The two volumes will not sync until one of them is set to be the primary.

On one of the nodes, force it to be the primary node.

drbdadm -- --force primary mysql

On the primary node, create a file system:

mkfs -t xfs /dev/drbd0

You can now mount the device on the primary node:

mount /dev/drbd0 /mnt/drbd

At this stage, you should be seeing the two volumes beginning to sync:

# drbd-overview
 0:mysql/0  SyncSource Primary/Secondary UpToDate/Inconsistent /mnt/drbd xfs 35G 34M 35G 1% 
	[=>..................] sync'ed: 14.4% (30480/35580)M

Setting up MySQL

Install MySQL (or MariaDB)

yum -y install mariadb mariadb-server

If you mounted the drbd volume to /mnt/drbd, you could make a symlink for /var/lib/mysql.

ln -s /mnt/drbd/mysql /var/lib/mysql

Initialize the database:

mysql_install_db 

Corosync / CRM

You'll need crmsh if you want to follow some other guides. Install it using the repository at:

http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/

cd
wget http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/python-pssh-2.3.1-4.2.x86_64.rpm http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/pssh-2.3.1-4.2.x86_64.rpm http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/crmsh-2.1.2-1.1.x86_64.rpm
yum install *.rpm

Configure the cluster so we have a an IP address set, MySQL running, and the DRBD volume mounted.

crm configure
primitive p_ip_mysql ocf:heartbeat:IPaddr2 \
  params ip="172.20.10.8" cidr_netmask="24" \
  op monitor interval="30s"
primitive p_drbd_mysql ocf:linbit:drbd \
  params drbd_resource="mysql" \
  op start timeout="90s" \
  op stop timeout="180s" \
  op promote timeout="180s" \
  op demote timeout="180s" \
  op monitor interval="30s" role="Slave" \
  op monitor interval="29s" role="Master"
primitive p_fs_mysql ocf:heartbeat:Filesystem \
  params device="/dev/drbd1" \
    directory="/mnt/drbd" \
    fstype="xfs" \
    options="relatime" \
  op start timeout="60s" \
  op stop timeout="180s" \
  op monitor interval="60s" timeout="60s"
primitive p_mysql ocf:heartbeat:mysql \
  params additional_parameters="--bind-address=172.20.10.8" \
    config="/etc/my.cnf" \
    pid="/var/run/mariadb/mariadb.pid" \
    socket="/var/lib/mysql/mysql.sock" \
    log="/var/log/mariadb/mariadb.log" \
  op monitor interval="20s" timeout="10s" \
  op start timeout="120s" \
  op stop timeout="120s"
group g_mysql p_ip_mysql p_fs_mysql p_mysql
ms ms_drbd_mysql p_drbd_mysql \
  meta notify="true" clone-max="2"
colocation c_mysql_on_drbd inf: g_mysql ms_drbd_mysql:Master
order o_drbd_before_mysql inf: ms_drbd_mysql:promote g_mysql:start
commit

We'll need to configure STONITH.

You can see all the available STONITH devices using:

stonith_admin -I

You can use any of the listed devices as part of your configuration. (Added after stonith:_________)

crm configure
 property stonith-enabled=true
 property stonith-action=poweroff
 rsc_defaults resource-stickiness=100
 property no-quorum-policy=ignore
 
 primitive vm-fence-c1 stonith:fence_vmware_soap \
  params ipaddr="172.17.12.8" login="root" passwd="xxx" \
    port="OpenStack C1" ssl="1" retry_on="20" shell_timeout="10" \
    login_timeout="10" action="off" verbose="true"
 
 primitive vm-fence-c2 stonith:fence_vmware_soap \
  params ipaddr="172.20.1.27" login="root" passwd="xxx" \
    port="OpenStack C2" ssl="1" retry_on="20" shell_timeout="10" \
    login_timeout="10" action="off" verbose="true"

 location l-vm-c1 vm-fence-c1 -inf: os-c1
 location l-vm-c2 vm-fence-c2 -inf: os-c2
commit

Other Commands

Change a node to a secondary on a specific resource (eg: mysql):

drbdadm secondary mysql

To show the current role of a node on a specific resource:

drbdadm role mysql

Split Brain

The two nodes will either be in the WFConnection or StandAlone state:

[root@os-c1 mnt]# drbd-overview 
 0:mysql/0  StandAlone Secondary/Unknown UpToDate/DUnknown 

[root@os-c2 mnt]# drbd-overview 
 0:mysql/0  StandAlone Primary/Unknown UpToDate/DUnknown 

Make both nodes in the StandAlone state by disconnecting the resource if it is in the WFConnection state.

Set the two nodes to secondary

drbdadm secondary mysql

The node which that is to have its data discarded will then need connect with the --discard-my-data option:

drbdadm connect --discard-my-data mysql

The node that is to have its data preserved should just connect:

drbdadm connect mysql

Both nodes should not be in the StandAlone state and should at this point be connected and syncing.

See Also

• https://drbd.linbit.com/users-guide/s-build-from-source.html
• http://unix-linux-server.blogspot.ca/2014/10/drbd-on-centos-70-oracle-linux-70-and.html
• http://wiki.centos.org/HowTos/Ha-Drbd (CentOS 5)
• http://wiki.openvz.org/HA_cluster_with_DRBD_and_Heartbeat