Nomad

HashiCorp Nomad

Cheat sheet

Task	Command
Start an agent `-dev` mode for testing	`nomad agent -dev -bind 0.0.0.0 -log-level INFO`
Show node status	`nomad node status`
Show connected members	`nomad server members`
Create an example job	`nomad job init`
Run a job	`nomad job run example.nomad`
Show a job's status	`nomad job status example`
Examine an allocation	`nomad alloc status allocation-id`
Examine an allocation's logs	`nomad alloc logs allocation-id task-name`

Concepts

See: https://learn.hashicorp.com/tutorials/nomad/get-started-vocab?in=nomad/get-started

Task: Smallest unit of work and is executed by a task driver. Eg. a container running via Docker
Group: A set of tasks that must run together. Eg. a web server and a database server
Job: A set of (one or more) groups. Used to constrain the scheduler, set update strategies, ACL

When defining a job, you:

Specify a job definition
Within the job, you specify one or more groups
Within each group, you specify its replica (count=#), task and service

Installation

Installation instructions are outlined at https://www.nomadproject.io/docs/install. For CentOS, run:

# yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# yum -y install nomad

You can start Nomad immediately by running nomad agent -dev-connect. In dev mode, you can poke around with nomad with minimal risk since nothing you do will persist.

Configuration

The Nomad configuration file is located at: /etc/nomad.d/nomad.hcl.

data_dir = "/opt/nomad/data"
bind_addr = "0.0.0.0"

server {
  enabled = true
  bootstrap_expect = 1
}

client {
  enabled = true
  servers = ["127.0.0.1:4646"]
}

But when you start Nomad, you need to specify it with -config: nomad agent -config=/etc/nomad.d. You can also specify multiple locations with additional -config parameters. A config path can also be a directory, whereby Nomad will then load all .hcl and .json files within that directory.

Question: Does this file need to be replicated on all member nodes? Or does it replicate automatically somehow as diagramed at https://learn.hashicorp.com/tutorials/nomad/get-started-vocab?in=nomad/get-started#nomad-cluster

Docker

If you intend to use docker volumes, you need to enable it by specifying a custom docker plugin config. An example is given below.

plugin "docker" {
    config {
        endpoint = "unix:///var/run/docker.sock"
        volumes {
            enabled      = true
            selinuxlabel = "z"
        }
    }
}

See: https://www.nomadproject.io/docs/drivers/docker#enabled-1

By default with Docker tasks, network namespaces between tasks within a group are isolated. You can have multiple containers be on the same network namespace by specifying within group -> network, mode = "bridge". This is useful if you were to run a web service application with a database backend but do not necessarily want to expose the database port.

Other

Consul integration

Start consul: consul agent -dev

Nomad appears to just detect this automatically, like magic.

Persistent Volumes

Host Volumes

To have a container mount a specific host volume, we need to:

Define a host_volume stanza in the Nomad agent client configuration. Restart the Nomad agent to apply.
Define a volume declaration within a task's group.
Define a volume_mount declaration in a task.

If you are using Docker, you may be interested in skipping all this and just mounting the volumes as you would with docker-compose in the sub section below.

Step 1: Define a host_volume stanza within the client block in your Nomad configuration file. See: https://www.nomadproject.io/docs/configuration/client#host_volume-stanza

host_volume "mysql" {
    path      = "/opt/mysql/data"
    read_only = false
  }

Restart Nomad and then verify that the new host volume shows up when running nomad node status -short -self.

Step 2: Within the job/group, define the volume. Eg:

group "something" {
  volume "mysql" {
    type = "host"
    source = "mysql"
    read_only = false
  }
  ...
}

Step 3: Within the task, mount this volume:

group "something" {
  ...
  task "container" {
    ...
    volume_mount {
        volume      = "mysql"
        destination = "/var/lib/mysql"
        read_only   = false
    }
    ...
  }
}

The downside/upside to this way of doing things is that the volume is tied to the Nomad agent (which is ideal if you want to only schedule this task on nodes that have the volume available), but adding additional host_volumes requires restarting the agent.

Docker side note

If you have enabled Docker volumes, you can also add additional volumes without needing to define host_volumes simply by listing the volumes within the config section of the task. For example, to mount /local/volumes/container1 to /data in the container:

task "container" {
  driver = "docker"

  config {
	image = "..."

	volumes = [
		"/local/volumes/container1:/data"
	]
...
}

NFS Volumes

I am trying to replicate what I had with docker-compose where certain volumes were automatically mounted via NFS using the ContainX/docker-volume-netshare Docker plugin. Combing through the documentation and through some trial and error, I found out that you can define a specific volume driver to use with Docker.

To get this working, install the docker-volume-netshare plugin and start the plugin as a service.

In a task definition, specify a mount configuration as follows.

task "something" {
  driver = "docker"

  config {
	image = "some-image"
	ports = ["http"]

	mount {
		type = "volume"
		target = "/TV"
		source = "tv-volume"
		# readonly = true

		volume_options {
			driver_config {
				name = "nfs"
				options {
					share = "bnas:/remote/nfs/export"
				}
			}
		}
	}

  }
  
  ...
}

The target is the path of the volume in the container. The NFS server and export are defined as a the share in the driver option. The mount source value is analogous to the volume name you specify in the volumes definition in docker-compose.yml and is the directory name that netshare uses to mount the volume on the host machine. In the example above, when the container starts, it netshare will mount the volume to /var/lib/docker-volumes/netshare/nfs/tv-volume.

If you get a volumes are not enabled; cannot mount host paths: error, you need to enable docker volumes in the docker plugins configuration. Edit nomad.hcl and add:

plugin "docker" {
    config {
        endpoint = "unix:///var/run/docker.sock"
        volumes {
            enabled      = true
            selinuxlabel = "z"
        }
    }
}

Traefik load balancing

See: https://learn.hashicorp.com/tutorials/nomad/load-balancing-traefik

Configure Traefik with the consulCatalog provider in the traefik.toml configuration file:

# Enable Consul Catalog configuration backend.
[providers.consulCatalog]
    prefix           = "traefik"
    exposedByDefault = false

    [providers.consulCatalog.endpoint]
      address = "127.0.0.1:8500"
      scheme  = "http"

When defining a service within a job in Nomad, specify the following tags:

service {
      name = "demo-webapp"
      port = "http"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.http.rule=Path(`/myapp`)",
      ]

Traefik will then be notified of the new backends when changes to this service happens via Consul.

Open Questions

Can we have the load balancing Traefik instance be an external service? If so, how?