CloudStack

Apache CloudStack is open-source cloud computing software. It is used to deploy a infrastructure as a service (IaaS) platform on virtualization technologies such as KVM, VMware, and Xen. This is similar to OpenStack but is significantly simpler to setup and manage (albeit with less features).

This page contains my notes on setting up and using CloudStack 4.15. I am by no means a CloudStack expert so take my notes here with a huge grain of salt and feel free to make corrections.

Installation

The installation is based on CloudStack 4.15 using CentOS 8. The setup described below uses KVM and Open vSwitch. I'm basing the design decisions and approach from the installation guide at http://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html

Overview

I will have 1 management node and a few bare metal nodes. All nodes will have the same processor (Intel something) and memory (24GB).

Each node will have the same network configuration based on OpenVSwitch. There will be only 1 ethernet connection per node with various VLANs trunked to each node. The VLANs are:

1. management (vlan 11, untagged), 172.19.0.0/20
2. storage (vlan 3205), 172.22.0.0/24
3. guest (vlan 100-200)
4. public (vlan 2), 136.159.*.0/24

Node setup

Each node will be set up with the following sub-steps.

CloudStack Repos

Install CloudStack repos.

# cat > /etc/yum.repos.d/cloudstack.repo <<EOF
[cloudstack]
name=cloudstack
baseurl=http://download.cloudstack.org/centos/8/4.15/
enabled=1
gpgcheck=0
EOF

Install base packages

Install all other dependencies.

# yum -y install epel-release
# yum -y install bridge-utils net-tools

Install OpenVSwitch from CentOS Extras:

# yum -y install \
http://mirror.centos.org/centos/8/extras/x86_64/os/Packages/centos-release-nfv-openvswitch-1-3.el8.noarch.rpm \
http://mirror.centos.org/centos/8/extras/x86_64/os/Packages/centos-release-nfv-common-1-3.el8.noarch.rpm

Disable SELinux

The system should have SELinux disabled. Use setenforce and edit the selinux config:

# setenforce 0
# vi /etc/selinux/config 
## disable selinux

Disable firewalld

# systemctl stop firewalld
# systemctl disable firewalld

Configure Open vSwitch

# echo "blacklist bridge" >> /etc/modprobe.d/local-blacklist.conf
# echo "install bridge /bin/false" >> /etc/modprobe.d/local-dontload.conf

# systemctl start openvswitch
# systemctl enable openvswitch

We will be using network-scripts to configure the Open vSwitch bridges later. I removed NetworkManager but retained network-scripts to ensure NetworkManager doesn't interfere with my network setup. The install guide leaves NetworkManager around.

I create a 'shared' bridge that's tied to the network interface called nic0. This was done to make it easier to change the bridge setup during my testing but this could be simplified. Each of the physical networks I later set up in CloudStack are its own individual bridge to make it obvious how VMs get connected to the network.

# ovs-vsctl add-br   nic0
# ovs-vsctl add-port nic0 enp4s0f0 tag=11 vlan_mode=native-untagged
# ovs-vsctl set port nic0 trunks=2,11,40-49,3205

# ovs-vsctl add-br management0 nic0 11
# ovs-vsctl add-br cloudbr0 nic0 2
# ovs-vsctl add-br cloudbr1 nic0 100
# ovs-vsctl add-br storage0 nic0 3205

The node's management IP address needs to be removed from the primary network interface and then assigned on the management0 interface. If you're doing this to a node remotely, this might interrupt your connection.

# ip addr del 172.19.12.141/20 dev enp4s0f0
# ip addr add 172.19.12.141/20 dev management0
# ip route add default via 172.19.0.3
# ip addr add 172.22.0.241/24 dev storage0

# ip link set management0 up
# ip link set storage0 up

Network configuration

Setup networking using network-scripts. The idea here is to have the Open vSwitch network bridges come up on boot with the appropriate static IP adresses. In this setup, I only provided static IP addresses for both the management and storage networks but left the public and guest networks unconfigured. All the bridges should be configured to come up on boot however. For bridges that require a static IP, I used something like the following:

# cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPV6INIT=no
IPV6_AUTOCONF=no
DELAY=5
IPADDR=172.16.10.2
GATEWAY=172.16.10.1
NETMASK=255.255.255.0
DNS1=8.8.8.8
DNS2=8.8.4.4
USERCTL=no
NM_CONTROLLED=no
EOF

For bridges that don't require an IP, I used something like this:

cat <<EOF > ifcfg-cloudbr0
DEVICE=cloudbr0
TYPE=OVSBridge
DEVICETYPE=ovs
ONBOOT=yes
BOOTPROTO=none
HOTPLUG=no
NM_CONTROLLED=no
EOF

In my setup, I configured the following interfaces on each of the nodes:

Network Interface	Role	Configuration
enp4s0f0	primary NIC in the host	up on boot; no IP
nic0	network OVS switch that connects to the other bridges to the NIC	up on boot; no IP
cloudbr0	public traffic.	up on boot; no IP
cloudbr1	guest traffic	up on boot; no IP
management0	management traffic	up on boot; assigned with management IP
storage0	storage traffic	up on boot; assigned with storage network IP

Management node setup

On the management node, set up the network configs and the CloudStack management packages.

Setup Storage

If you intend to use the management server as the primary and secondary storage, you will need to set up a NFS server. If you intend to use an external NFS server as the primary storage, you can skip this step.

# mkdir -p /export/primary /export/secondary
# yum -y install nfs-utils
# cat > /etc/exports <<EOF
/export/secondary *(rw,async,no_root_squash,no_subtree_check)
/export/primary *(rw,async,no_root_squash,no_subtree_check)
EOF
# systemctl start nfs-server
# systemctl enable nfs-server

CloudStack management services

Install MySQL. MariaDB isn't supported and the installation fails with it.

# rpm -ivh http://repo.mysql.com/mysql80-community-release-el8.rpm
# yum -y install mysql-server
# yum -y install mysql-connector-python

## edit /etc/my.cnf to have the following lines.
cat >> /etc/my.cnf <<EOF
[mysqld]
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'
EOF

# systemctl enable mysqld
# systemctl start mysqld

Setup CloudStack.

# yum -y install cloudstack-management

# cloudstack-setup-databases cloud:password@localhost --deploy-as=root
# cloudstack-setup-management
# systemctl start cloudstack-management
# systemctl enable cloudstack-management

You will need to seed the systemvm images to the secondary storage. If you are using an external NFS server for your secondary storage, adjust the mount point in the command accordingly.

## Seed the systemvm into secondary storage
# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /export/secondary -u https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2 -h kvm -F

We will continue the setup process via the web interface after setting up a bare metal node.

Bare metal node setup

You should set up at least one bare metal node which will be used to set up your first zone and pod.

On a bare metal node, set up everything outlined in the Node setup section above. The node should have the CloudStack repos, Open vSwitch, SElinux/firewalld, and the networking configured. The agent node must have virtualization enabled on the CPU and KVM should be installed. You should be able to find /dev/kvm on the system. To set up the node, install the cloudstack-agent package.

# yum -y install cloudstack-agent

Configure qemu and libvirtd.

## edit /etc/libvirt/qemu.conf 
vnc_listen=0.0.0.0

## edit /etc/libvirt/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
mdns_adv = 0

The CloudStack install guide instructs you to edit the libvirtd arguments to --listen, but this will prevent libvirtd from starting using systemd. Instead, you should skip this step entirely because the CloudStack agent will configure this for you when you add the node to a zone.

## The install guide suggests editing /etc/sysconfig/libvirtd to use the listen flag.
## However, this only works if you're not using systemd or using the libvirtd-tcp socket.
## I skipped this step since the agent will configure this later on.
LIBVIRTD_ARGS="--listen"

Setup CloudStack

At this point in the process, your management node should be up and running and it should be serving the CloudStack web UI at http://cloudstack:8080/client. Login using the default admin / password credentials.

You will be greeted with a setup wizard. I have had no luck with this and it's better to ignore it. Instead, navigate to zones and manually set up your first zone.

Configuration

Zone types and security groups

There are 3 types of zones that you can create:

1. Basic zone - where all guest VMs are placed on a flat network.
2. Advanced zone - where guest VMs can be placed on individual VLAN based isolated networks. Each isolated network will have a virtual router (VR) which provides NAT/SNAT and firewall services against one or more public IP addresses. This is the only zone which allows creation of VPCs and isolated guest networks.
3. Advanced zone with security groups - where guest VMs are placed on a shared network that is publicly routable. There is no concept of a 'public' network because the guest network should also be public. As a result, there is no ability to create any other kind of guest networks (such as isolated networks with (S)NAT). The only benefit here is the ability to define security groups per-VM (which is implemented via IPTables on the bare metal host).

Because enabling security groups in a zone will restrict that zone from being able to create isolated guest networks, the security group feature only appears useful in an environment where guests only need to connect to the internet.

Open-ended questions

Console Proxy via DNS name?

Is it possible to set up console proxy to use a specific DNS name rather than directly using the public IP address that the console proxy was assigned?

This is possibly an issue if the console proxy and web services need to sit behind a reverse proxy (yes, I know, they should have a public IP already).

Compute offerings with 'unlimited' CPU cycles?

Compute offerings require a MHz value assigned. Why is this? Can we just assign a VM entire cores?

Tools

CloudMonkey

Download from: https://github.com/apache/cloudstack-cloudmonkey/releases/tag/6.1.0

Task	Command
Create a network	create network displaytext=ExternalNetwork name=ExternalNetwork networkofferingid=DefaultSharedNetworkOfferingId zoneid=ZoneId startip=x.x.x.x endid=x.x.x.y netmask=255.255.255.0 vlan=vlanid

Troubleshooting

When you run into issues, check the logs in /var/log/cloudstack/. There's typically a stacktrace which gets generated whenever you encounter an error.

Can't create shared network in a advanced zone using Open vSwitch

Whenever I try creating a shared network in an advanced zone that is using OVS, the step fails with: "Unable to convert network offering with specified id to network profile"

The OVS Guru appears to fail at designing the network which results in this failure. I'm still not sure why.