Firecracker

Firecracker is a virtual machine monitor that is capable of quickly starting Linux guest VMs using KVM. It's main selling feature is the speed to start up VMs. It is developed by AWS and is used by AWS in their serverless Lambda and Fargate offerings.

Setup

You need to have a Linux host with KVM already installed. The host must have a CPU should have virtualization flags which you can verify by running cat /proc/cpuinfo | grep -iE 'vmx|svm'. Intel uses the vmx flag while AMD uses the svm flag. If you intend to run Firecracker as a normal user, ensure that /dev/kvm is readable by that user.

Build or download the binaries from their Github repo: https://github.com/firecracker-microvm/firecracker

Building

# docker run --rm -ti -v `pwd`/firecracker:/firecracker rust:1.49 sh
## Inside the container, run
# git clone https://github.com/firecracker-microvm/firecracker.git

## Build, as per the README
# cd firecracker/
# tools/devtool build
# toolchain="$(uname -m)-unknown-linux-musl"

Introduction

Firecracker comes with two components: Firecracker which is the virtual machine monitor (VMM) itself, and a separate jailer program, which runs Firecracker inside a dedicated namespace and optionally limits what system calls can be used. You can make use of Firecracker without the jailer component.

The reason for Firecracker's speed is its minimalistic approach. It eliminates extraneous devices and the BIOS by only providing network, block, serial, and a keyboard (for shutdown) device support only.

Running VMs

Firecracker can be interacted via a HTTP API using the API socket that each instance sets up. The API is optional and can be disabled using the --no-api flag.

Basic Setup

The Firecracker README has some instructions on how to get your first VM running, which I will summarize below:

# Obtain the kernel and initramfs
$ wget https://s3.amazonaws.com/spec.ccfc.min/img/hello/kernel/hello-vmlinux.bin
$ wget https://s3.amazonaws.com/spec.ccfc.min/img/hello/fsfiles/hello-rootfs.ext4

# Create a config file outlining the VM
$ cat <<EOF > config.json
{
  "boot-source": {
    "kernel_image_path": "hello-vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "hello-rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 1024,
    "ht_enabled": false
  }
}
EOF

# Start the VM
$ firecracker --no-api --config-file config.json

The VM should boot into a functional Linux system in a few seconds. The guest OS serial console is mapped to Firecracker's standard input/output. To stop the VM, I had to kill Firecracker from another terminal.

With Networking

Building on the example in the previous section, we can add networking to the guest system via a tun/tap interface on the host.

On the host:

# ip tuntap add dev tap0 mode tap
# sysctl -w net.ipv4.conf.tap0.proxy_arp=1
# sysctl -w net.ipv6.conf.tap0.disable_ipv6=1
# ip addr add 172.17.0.1/16 dev tap0
# ip link set tap0 up

Configure the VM to have this network interface with the network-interfaces section. The MAC address can be specified with guest_mac but can be left undefined for random MAC address.

{
  "boot-source": {
    "kernel_image_path": "hello-vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "hello-rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "network-interfaces": [
      {
          "iface_id": "eth0",
          "host_dev_name": "tap0"
      }
  ],
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 1024,
    "ht_enabled": false
  }
}

After starting up the VM again, you should see that eth0 is up but without an address. You will need to bring the interface up manually:

# ip addr add 172.17.100.1/16 dev eth0
# ip link set eth0 up

Or alternatively, pass the interface config as a kernel parameter. See: https://mjmwired.net/kernel/Documentation/filesystems/nfs/nfsroot.txt#81. In general, you pass: ip=IP_ADDRESS::GATEWAY:NETMASK:HOSTNAME:DEVICE:AUTOCONFIG, where AUTOCONFIG should be set to off for a static network configuration. Change the kernel arguments in the config.json file:

...
  "boot-source": {
    "kernel_image_path": "hello-vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off ip=172.17.100.1::172.17.0.1:255.255.0.0::eth0:off"
  },
...

You should be able to ping the gateway 172.17.0.1, which will be the tap0 device on the host system.

How about internet access?

You need to make your host route traffic and act as a NAT. You then need to specify the nameservers for the guest system.

Tasks

Custom Firecracker Images

See: https://github.com/firecracker-microvm/firecracker/blob/master/docs/rootfs-and-kernel-setup.md

Troubleshooting

Error: Internal(KvmContext(KvmCap(Xcrs)))

I tried to run Firecracker inside a VM under Proxmox (KVM based) but it kept on throwing this error:

Building VMM configured from cmdline json failed: Internal(KvmContext(KvmCap(Xcrs)))

For some reason, KVM doesn't have the extended control registers? Not sure why this is...

Error: Internal(Vm(VmFd(Error(12))))

I tried to run Firecracker inside a tiny KVM based VPS and sometimes got:

Building VMM configured from cmdline json failed: Internal(Vm(VmFd(Error(12))))

Error 12 is when it isn't able to allocate memory. This is likely your machine running low on memory.