Terraform

Terraform is a "infrastructure as code" deployment tool. In a nutshell, Terraform allows you to define virtual machine configurations in HashiCorp Configuration Language (HCL) and then have Terraform do all the work of setting up these VMs for you automatically. The benefit of this approach is that your configs are declarative and any changes can be version controlled, are repeatable (idempotent too), and predictable.

Quick usage guide

Installation

Refer to the documentation at: https://www.terraform.io/downloads

Ubuntu

# curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
# apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
# apt-get update && sudo apt-get install terraform

Red Hat / Fedora / Rocky Linux

# dnf install -y dnf-plugins-core
## Fedora
# dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
## Red Hat / Rocky Linux / CentOS
# dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# dnf -y install terraform

Build from source

Terraform workflow

When working with Terraform, there are 4 actions that you will be using.

First, you need to initialize your providers -- the plugins that you will need to interact with your underlying infrastructure. This is accomplished by creating a main.tf file defining your providers and then running terraform init.

Next, define your resources in the same main.tf file. Your infrastructure's networking, virtual machines, etc. will be defined in this file. Once you're ready, run terraform plan to view what the changes Terrafom will perform. If everything looks good, run terraform apply to create your resources.

Finally, you can tear down everything using terraform destroy.

In summary:

Description	Command
Initialize and setup any providers defined in your configuration.	`terraform init`
Show any changes Terraform will do	`terraform plan`
Apply the changes (create/modify/destroy) to VMs as required.	`terraform apply`
Tear down everything	`terraform destroy`

Provider-specific notes

Below are some quick notes on specific IaaS platforms to help get you started with Terraform. There are a plethora of different providers that you can find more about on Terraform's documentation: https://runebook.dev/en/docs/terraform/-index-#Providers

Proxmox

There are a number of Proxmox providers for Terraform. The most popular one is Telmate/proxmox which will be the one we'll be using here. For more information on this provider and to see all the available input parameters, see:https://registry.terraform.io/providers/Telmate/proxmox/latest/docs/resources/vm_qemu.

Create a main.tf file with the following:

# Use Telmate proxmox provider
terraform {
  required_providers {
    proxmox = {
      source = "telmate/proxmox"
      version = "2.7.4"
    }
  }
}

# Configure the proxmox provider
provider "proxmox" {
  pm_api_url = "https://proxox-server:8006/api2/json"
  pm_api_token_id = "terraform@pam!terraform_token_id"
  pm_api_token_secret = "e458e7bc-d8e6-4028-885b-d0896f4becfa"
  pm_tls_insecure = true
}

Setup an account with an API key on Proxmox. This account should have permissions to '/' as well as to any storage volumes it needs to create VMs on '/storage/data'. (TBD)

Run terraform init to set up the provider.

Deploy a VM

After setting up the Proxmox provider, add the following to your main.tf file.

# Define our first VM
resource "proxmox_vm_qemu" "test-vm" {
  count = 1                            # 0 will destroy the VM
  name = "test-vm-${count.index + 1}"  # count.index starts at 0. We want the VM to be named test-vm-1
  target_node = var.proxmox_host       # target proxmox host
  clone = var.template_name            # VM template to clone from
  
  os_type = "cloud-init"
  agent = 1
  cores = 2
  sockets = 1
  cpu = "host"
  memory = 2048
  scsihw = "virtio-scsi-pci"
  bootdisk = "sata0"

  disk {
    slot = 0
    size = "10G"
    type = "sata"
    storage = "data"
    iothread = 1
  }
  
  network {
    model = "virtio"
    bridge = "vmbr0"
  }
  
  lifecycle {
    ignore_changes = [
      network,
    ]
  }
  
  ipconfig0 = "ip=10.1.1.10${count.index + 1}/22,gw=10.1.1.1"
  
  # sshkeys set using variables. the variable contains the text of the key.
  sshkeys = <<EOF
  ${var.ssh_key}
  EOF
}

vars.tf:

variable "ssh_key" {
  default = "ssh-rsa  ... "
}

# This should be the exact same name as your proxmox node name
variable "proxmox_host" {
	default = "proxmox-server"
}

variable "template_name" {
	default = "rocky85-template"
}

Run terraform plan to plan the tasks
Run terraform apply to create the VMs.

CloudStack

Create a main.tf file with the following:

terraform {
  required_providers {
    cloudstack = {
      source = "cloudstack/cloudstack"
      version = "0.4.0"
    }
  }
}

provider "cloudstack" {                                                                                                                          
  api_url    = "${var.cloudstack_api_url}" 
  api_key    = "${var.cloudstack_api_key}" 
  secret_key = "${var.cloudstack_secret_key}" 
}

Create a vars.tf with the following:

variable "cloudstack_api_url" { 
   default = "http://cloudstack-management:8080/client/api" 
} 
variable "cloudstack_api_key" { 
   default = " 
} 
variable "cloudstack_api_secret_key" {                                                                                                           
   default = "" 
}

Ensure that your API URL ends with /client/api. If you don't already have a API key and secret key, log in to the CloudStack console and navigate to your profile. Click on 'Generate Keys' and copy down the keys listed under your profile. Populate the values into vars.tf.

Run terraform init to set up the provider.

Create your first network and VM

Here's a Terraform file that sets up a VPC, guest network, a network ACL, and a few VMs using a custom Cloud Init payload.

# Create a new VPC
resource "cloudstack_vpc" "default" {
  name = "rcs-vpc"
  display_text = "rcs-vpc"
  cidr = "100.64.0.0/20"
  vpc_offering = "Default VPC offering"
  zone = "zone1"
}

# Create a new ACL
resource "cloudstack_network_acl" "default" {
    name  = "vpc-acl"
    vpc_id = "${cloudstack_vpc.default.id}"
}

# One ingress and one egress rule for the ACL
resource "cloudstack_network_acl_rule" "ingress" {
    acl_id = "${cloudstack_network_acl.default.id}"

    rule {
        action       = "allow"
        cidr_list    = ["10.0.0.0/8"]
        protocol     = "tcp"
        ports        = ["22", "80", "443"]
        traffic_type = "ingress"
    }
}

resource "cloudstack_network_acl_rule" "egress" {
    acl_id = "${cloudstack_network_acl.default.id}"

    rule {
        action       = "allow"
        cidr_list    = ["0.0.0.0/0"]
        protocol     = "all"
        traffic_type = "egress"
    }
}


# Create a new network in the VPC
resource "cloudstack_network" "leosnet" {
    name = "leosnet"
    display_text = "leosnet"
    cidr = "100.64.1.0/24"
    network_offering = "DefaultIsolatedNetworkOfferingForVpcNetworks"
    acl_id = "${cloudstack_network_acl.default.id}"
    vpc_id = "${cloudstack_vpc.default.id}"
    zone = "zone1"
}

# Create a new public IP address for this network
resource "cloudstack_ipaddress" "public_ip" {
    vpc_id = "${cloudstack_vpc.default.id}"
    network_id = "${cloudstack_network.leosnet.id}"
}

# Create a port forwarding for SSH to the first VM we create
resource "cloudstack_port_forward" "ssh" {
    ip_address_id = "${cloudstack_ipaddress.public_ip.id}"

    forward {
        protocol  = "tcp"
        private_port = 22
        public_port = 22
        virtual_machine_id = "${cloudstack_instance.leo[0].id}"
    }
}

# Create VMs. We can create multiples by specifying count=
resource "cloudstack_instance" "leo" {
  count = 3
  name = "leo${count.index+1}"
  zone = "zone1"
  service_offering = "rcs.c4"
  # This template was created by Packer with CloudInit support
  template = "RockyLinux 8.5"
  network_id = "${cloudstack_network.leosnet.id}"

  # Warning: Enabling this option will reseult in VM's disks being deleted when the VMs are destroyed.
  # This option only works if 'allow.user.expunge.recover.vm' is set to true in global settings
  expunge = true

  user_data = <<EOF
#cloud-config
disable_root: false
chpasswd:
  list: |
    root:password
  expire: false

EOF
}

Run terraform plan to plan the tasks
Run terraform apply to create the VMs.

Using custom service offerings

If you wish to use a custom service offering, you will have to pass in the CPU and memory amount. The documentation isn't clear how this is actually done, but if you look in the deploy VirtualMahcine API call, you will see that the CPU and memory values are passed in via the 'details' parameter. This is supported by the Terraform provider and can be used like so:

resource "cloudstack_instance" "testing" { 
  name = "testing" 
  zone = "zone1" 
  service_offering = "icelake.custom" 
  details = { 
    "cpuNumber" = "1"                                                                                                                                                                                                                                                         
    "memory" = "1024" 
  }   
  template = "RockyLinux 9.4"
...
}

Troubleshooting

Issues with the apply step

I kept on getting "400 Parameter verification failed":

╷
│ Error: 400 Parameter verification failed.
│
│   with proxmox_vm_qemu.test-vm[0],
│   on main.tf line 20, in resource "proxmox_vm_qemu" "test-vm":
│   20: resource "proxmox_vm_qemu" "test-vm" {
│
╵

The issues I encountered which triggered this error was:

Incorrect host name for the Proxmox node
Invalid values for the disk type (I tried changing type = "scsi" to type = "sata") while leaving iothread = 1 defined.

To help diagnose issues, review the provider's documentation and ensure the values you're using are appropriate. You may also want to enable additional logging by running with the TF_LOG=TRACE environment variable:

$ TF_LOG=TRACE terraform apply