DRBD

From Leo's Notes
Last edited on 1 September 2019, at 06:22.

DRBD (Distributed Replicated Block Device) is a virtual block device that is distributed across two or more machines.

Installation[edit | edit source]

On a clean installation of RHEL 7 or equivalent, run updates, and ensure that you have NTP and a recent kernel installed.

From ElRepo[edit | edit source]

Get the DRBD packages from ELRepo: 

# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
# yum install drbd84-utils kmod-drbd84

You should be able to load the drbd module: 

# modprobe drbd
# lsmod

From Source[edit | edit source]

Install prerequisites: 

# yum install gcc flex make autoconf m4

Get the source from their git repository. Releases from tarballs may fail depending on which kernel version you're using. 

# git clone git://git.drbd.org/drbd-8.4.git
# cd drbd-8.4
# make
# make install

The make install should have loaded the module and you can verify this: 

# lsmod

If it isn't loaded, load it using modprobe 

# modprobe drbd

Compile the userspace utilities: 

# git clone git://git.drbd.org/drbd-utils.git
# cd drbd-utils
# ./aclocal.sh
# ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc
# make
# make install

Note: I got errors about man pages not being found. Ugh. Use the RPMs instead (it worked after a clean install...)

Configuration[edit | edit source]

This section will configure DRBD for use in a redundant MySQL setup.

DRBD syncs volumes. If you did not configure a partition or have a separate disk to mirror, you can create a file volume instead.

On all nodes: 

# mkdir /var/drbd
# cd /var/drbd
 
## Create a 50GB volume
# dd if=/dev/zero of=data_volume.dat bs=1024 count=52428800

## Create a partition
# echo -e "n\n\n\n\nw\nq\n"

On all nodes, configure the DRBD config in /etc/drbd.d/*conf. You may want to disable the survey.

On all nodes, configure DRBD by creating a file in /etc/drbd.d/mysql.res with: 

resource mysql {
	disk {
		on-io-error detach;
	}

	device    /dev/drbd0;
	disk      /dev/mapper/loop0p1;
	meta-disk internal;

	on $hostname {
		address 172.20.10.10:7700;
	}

	on $hostname {
		address 172.20.10.9:7700;
	}
}

TODO: Can you just use /dev/loop0 instead?!

Be sure to rename the hostnames so that it matches the actual machine's hostname. Otherwise, when you try creating the resource, you will get an error like: 

'mysql' not defined in your config (for this host).

On all nodes, create the resource: 

# drbdadm -v create-md mysql

If it complains about finding a filesystem, zero out the first few blocks using dd and try again or ignore and continue.

On all nodes, start the resource: 

# drbdadm up mysql

You can check the status of the drbd volume: 

# drbd-overview
0:mysql/0  Connected Secondary/Secondary Inconsistent/Inconsistent

The two volumes will not sync until one of them is set to be the primary.

On one of the nodes, force it to be the primary node. 

# drbdadm -- --force primary mysql

On the primary node, create a file system: 

# mkfs -t xfs /dev/drbd0

You can now mount the device on the primary node: 

# mount /dev/drbd0 /mnt/drbd

At this stage, you should be seeing the two volumes beginning to sync: 

# drbd-overview
 0:mysql/0  SyncSource Primary/Secondary UpToDate/Inconsistent /mnt/drbd xfs 35G 34M 35G 1% 
	[=>..................] sync'ed: 14.4% (30480/35580)M


Setting up MySQL[edit | edit source]

Install MySQL (or MariaDB) 

# yum -y install mariadb mariadb-server

If you mounted the drbd volume to /mnt/drbd, you could make a symlink for /var/lib/mysql. 

# ln -s /mnt/drbd/mysql /var/lib/mysql

Initialize the database: 

# mysql_install_db


Corosync / CRM[edit | edit source]

You'll need crmsh if you want to follow some other guides. Install it using the repository at: 

# cd
# wget http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/python-pssh-2.3.1-4.2.x86_64.rpm http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/pssh-2.3.1-4.2.x86_64.rpm http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-7/x86_64/crmsh-2.1.2-1.1.x86_64.rpm
# yum install *.rpm

Configure the cluster so we have a an IP address set, MySQL running, and the DRBD volume mounted. 


# crm configure
# primitive p_ip_mysql ocf:heartbeat:IPaddr2 \
   params ip="172.20.10.8" cidr_netmask="24" \
   op monitor interval="30s"
# primitive p_drbd_mysql ocf:linbit:drbd \
   params drbd_resource="mysql" \
   op start timeout="90s" \
   op stop timeout="180s" \
   op promote timeout="180s" \
   op demote timeout="180s" \
   op monitor interval="30s" role="Slave" \
   op monitor interval="29s" role="Master"
# primitive p_fs_mysql ocf:heartbeat:Filesystem \
   params device="/dev/drbd1" \
     directory="/mnt/drbd" \
     fstype="xfs" \
     options="relatime" \
   op start timeout="60s" \
   op stop timeout="180s" \
   op monitor interval="60s" timeout="60s"
# primitive p_mysql ocf:heartbeat:mysql \
   params additional_parameters="--bind-address=172.20.10.8" \
     config="/etc/my.cnf" \
     pid="/var/run/mariadb/mariadb.pid" \
     socket="/var/lib/mysql/mysql.sock" \
     log="/var/log/mariadb/mariadb.log" \
   op monitor interval="20s" timeout="10s" \
   op start timeout="120s" \
   op stop timeout="120s"
# group g_mysql p_ip_mysql p_fs_mysql p_mysql
# ms ms_drbd_mysql p_drbd_mysql \
   meta notify="true" clone-max="2"
# colocation c_mysql_on_drbd inf: g_mysql ms_drbd_mysql:Master
# order o_drbd_before_mysql inf: ms_drbd_mysql:promote g_mysql:start
# commit

We'll need to configure STONITH.

You can see all the available STONITH devices using: 

# stonith_admin -I

You can use any of the listed devices as part of your configuration. (Added after stonith:_________) 

# crm configure
# property stonith-enabled=true
# property stonith-action=poweroff
# rsc_defaults resource-stickiness=100
# property no-quorum-policy=ignore
  
# primitive vm-fence-c1 stonith:fence_vmware_soap \
   params ipaddr="172.17.12.8" login="root" passwd="xxx" \
     port="OpenStack C1" ssl="1" retry_on="20" shell_timeout="10" \
     login_timeout="10" action="off" verbose="true"
  
# primitive vm-fence-c2 stonith:fence_vmware_soap \
   params ipaddr="172.20.1.27" login="root" passwd="xxx" \
     port="OpenStack C2" ssl="1" retry_on="20" shell_timeout="10" \
     login_timeout="10" action="off" verbose="true"
 
# location l-vm-c1 vm-fence-c1 -inf: os-c1
# location l-vm-c2 vm-fence-c2 -inf: os-c2
# commit

Other Commands[edit | edit source]

Change a node to a secondary on a specific resource (eg: mysql): 

# drbdadm secondary mysql

To show the current role of a node on a specific resource: 

# drbdadm role mysql

Split Brain[edit | edit source]

The two nodes will either be in the WFConnection or StandAlone state: 

[root@os-c1 mnt]# drbd-overview 
0:mysql/0  StandAlone Secondary/Unknown UpToDate/DUnknown 

[root@os-c2 mnt]# drbd-overview 
0:mysql/0  StandAlone Primary/Unknown UpToDate/DUnknown

Make both nodes in the StandAlone state by disconnecting the resource if it is in the WFConnection state.

Set the two nodes to secondary 

# drbdadm secondary mysql

The node which that is to have its data discarded will then need connect with the --discard-my-data option: 

# drbdadm connect --discard-my-data mysql

The node that is to have its data preserved should just connect: 

# drbdadm connect mysql

Both nodes should not be in the StandAlone state and should at this point be connected and syncing.

See Also[edit | edit source]


Various Linux Notes
Linux Tools and Utilites
Package Management
Linux Distributions
Networking
Containers