Nomad

HasiCorp Nomad

Cheat sheet

Allocations are like pods, it contains one or more 'tasks'

Tasks are just containers

Job files contain all the information about the deployment.

Task	Command
Start an agent	nomad agent -dev -bind 0.0.0.0 -log-level INFO
Show node status	nomad node status
Show connected members	nomad server members
Create an example job	nomad job init
Run a job	nomad job run example.nomad
Show a job's status	nomad job status example
Examine an allocation	nomad alloc status allocation-id
Examine an allocation's logs	nomad alloc logs allocation-id task-name

Concepts

See: https://learn.hashicorp.com/tutorials/nomad/get-started-vocab?in=nomad/get-started

• Task: Smallest unit of work and is executed by a task driver. Eg. a container running via Docker
• Group: A set of tasks that must run together. Eg. a web server and a database server
• Job: A set of (one or more) groups. Used to constrain the scheduler, set update strategies, ACL

When defining a job, you:

1. Specify a job definition
2. Within the job, you specify one or more groups
3. Within each group, you specify its replica (count=#), task and service

Installation

Installation instructions are outlined at https://www.nomadproject.io/docs/install. For CentOS, run:

# yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# yum -y install nomad

Configuration

The Nomad configuration file is located at: /etc/nomad.d/nomad.hcl.

data_dir = "/opt/nomad/data"
bind_addr = "0.0.0.0"

server {
  enabled = true
  bootstrap_expect = 1
}

client {
  enabled = true
  servers = ["127.0.0.1:4646"]
}

But when you start Nomad, you need to specify it with -config: nomad agent -dev-connect -config=/etc/nomad.d/nomad.hcl. You can also specify multiple locations with additional -config parameters. A config path can also be a directory, whereby Nomad will then load all .hcl and .json files within that directory.

Question: Does this file need to be replicated on all member nodes? Or does it replicate automatically somehow as diagramed at https://learn.hashicorp.com/tutorials/nomad/get-started-vocab?in=nomad/get-started#nomad-cluster

Docker

See: https://www.nomadproject.io/docs/drivers/docker#enabled-1

You may want to enable volumes

Other

Consul integration

Start consul: consul agent -dev

Nomad appears to just detect this automatically, like magic.

Persistent Volumes

Local Volumes

To have a container use a host volume:

1. Define a host_volume stanza in the Nomad agent client configuration. Restart the Nomad agent to apply.
2. Define a volume declaration within a task's group.
3. Define a volume_mount declaration in a task.

Step 1: Define a host_volume stanza within the client block in your Nomad configuration file. See: https://www.nomadproject.io/docs/configuration/client#host_volume-stanza

host_volume "mysql" {
    path      = "/opt/mysql/data"
    read_only = false
  }

Restart Nomad and then verify that the new host volume shows up when running nomad node status -short -self.

Step 2: Within the job/group, define the volume. Eg:

volume "mysql" {
    type = "host"
    source = "mysql"
    read_only = false
  }

Step 3: Within the task, define:

volume_mount {
        volume      = "mysql"
        destination = "/var/lib/mysql"
        read_only   = false
      }

NFS Volumes

I am trying to replicate what I had with docker-compose where certain volumes were automatically mounted via NFS using the ContainX/docker-volume-netshare Docker plugin. Combing through the documentation and through some trial and error, I found out that you can define a specific volume driver to use with Docker.

In a task definition, specify a mount configuration as follows:

task "something" {
  driver = "docker"

  config {
	image = "some-image"
	ports = ["http"]

	mount {
		type = "volume"
		target = "/TV"
		source = "tv-volume"
		# readonly = true

		volume_options {
			driver_config {
				name = "nfs"
				options {
					share = "bnas:/remote/nfs/export"
				}
			}
		}
	}

  }
  
  ...
}

The mount source value is analogous to the volume name you specify in the volumes definition in docker-compose.yml and is the directory name that netshare uses to mount the volume on the host machine. In the example above, when the container starts, it netshare will mount the volume to /var/lib/docker-volumes/netshare/nfs/tv-volume.

If you get a "volumes are not enabled" error, you need to enable docker volumes in the docker plugins configuration. Edit nomad.hcl and add:

plugin "docker" {
    config {
        endpoint = "unix:///var/run/docker.sock"
        volumes {
            enabled      = true
            selinuxlabel = "z"
        }
    }
}

Traefik load balancing

See: https://learn.hashicorp.com/tutorials/nomad/load-balancing-traefik

Configure Traefik with the consulCatalog provider in the traefik.toml configuration file:

# Enable Consul Catalog configuration backend.
[providers.consulCatalog]
    prefix           = "traefik"
    exposedByDefault = false

    [providers.consulCatalog.endpoint]
      address = "127.0.0.1:8500"
      scheme  = "http"

When defining a service within a job in Nomad, specify the following tags:

service {
      name = "demo-webapp"
      port = "http"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.http.rule=Path(`/myapp`)",
      ]

Traefik will then be notified of the new backends when changes to this service happens via Consul.

Open Questions

• Can we have the load balancing Traefik instance be an external service? If so, how?

Other Questions

• Can the docker driver run as specific users? As said in https://www.nomadproject.io/docs/job-specification/task#user, "Docker and rkt images specify their own default users." which is super crummy if true as you can't force containers to run as non-root.