Docker Compose

From Leo's Notes
Last edited on 25 December 2021, at 10:24.

Docker Compose is a program that simplifies the building of Docker applications by allowing you to define all the service components of the application in a single yaml file.

Unlike Docker Swarm or Kubernetes, it does not provide orchestration or any infrastructure level support around your application. However, it does simplify defining services that are intended to run on a single-node Docker node. Rather than starting Docker containers from the command line, where volumes and network linking can get complicated, the entire application can be laid out in a yaml file and the entire application can be deployed using the docker-compose command.

Cheat Sheet

Commands

Task Command
Start all services defined in a docker-compose.yml file. docker-compose up -d
Stop all services

including volumes

docker-compose downdocker-compose down -v
Restart all services docker-compose restart
Update containers and restart only changed images docker-compose pull && docker-compose up -d
View logs docker-compose logs

docker-compose logs -f (to follow)

List containers that are deployed docker-compose ps
View resource usage of running containers docker-compose top
Validate docker compose file docker-compose config

Service definitions

What How
New service 
version: '3.3'
services:
  service_one:
      ...
Specify image 
version: '3.3'
services:
  service_one:
      image: my-image:latest
Build the image 
version: '3.3'
services:
  service_one:
      build:
        context: ./build-directory
        dockerfile: Dockerfile
Restart policy 
version: '3.3'
services:
  service_one:
      restart: always
  • no - don't restart (default)
  • on-failure - restart only if container exit code is nonzero
  • unless-stopped - always restarts, except when docker host restarts or docker service stops.
  • always - always restart. Containers come up with docker service.
Specify uid/gid 
version: '3.3'
services:
  service_one:
      user: "1000:1000"

Container runs as a specific uid/gid. Useful for enforcing rootless containers.

Expose port 
version: '3.3'
services:
  service_one:
      expose:
        - "80"
        - "443"

Exposes ports visible to other containers in the stack.

Publish a port 
version: '3.3'
services:
  service_one:
      ports:
        - "80:8888"
        - "192.168.1.1:80:88888"

Publishes a port on the docker host

Add capabilities 
version: '3.3'
services:
  service_one:
      cap_add:
        - SYS_ADMIN
        - NET_ADMIN

Add capabilities to the container. Refer to the Linux source for other keys. Common ones include:

  • SYS_ADMIN - Perform a range of system administration operations.
  • NET_ADMIN - Perform various network-related operations
  • SYS_NICE - Allows changing process nice values (good for batch jobs in containers)
  • SYS_TIME - Allows access to system clock (good if you want NTP in a container)
NFS Volume

netshare plugin 

services:
  service_one:
    volumes:
	  - NetworkShare:/share
	  
volumes:
  NetworkShare:
    driver: nfs
    driver_opts:
      share: "nas:/network/share"

You need the docker-volume-netshare plugin installed for this to work.

NFS Volume

local driver only 

services:
  service_one:
    volumes:
	  - NetworkShare:/share
	  
volumes:
  NetworkShare:
    driver: local
    driver_opts:
      type: nfs
      o: "username=user,password=secret,addr=1.2.3.4,soft,rw"
      # Share path on your NAS:
      device: ":/share"
Samba / CIFS

Volume 

services:
  service_one:
    volumes:
	  - NetworkShare:/share
	  
volumes:
  NetworkShare:
    driver: local
    driver_opts:
      type: cifs
      o: "username=user,password=secret,rw"
      device: "//nas/share"

Installation

To use Docker Compose, download the latest binary and stick it in your PATH. Get the binary from:

Example usage

For instance, if a LAMP stack is required, a docker-compose configuration file would define each of the components making up the LAMP stack (Apache+PHP, MySQL, optionally a Load Balancer). Rather than manually creating/removing each of the Docker containers and networks manually, the entire service can be brought up or down with one command.

An example docker-compose.yml file containing such a LAMP stack:

version: '3.3'

services:
  traefik:
    image: traefik:latest
    restart: always
    command: --web --docker --docker.watch --docker.exposedbydefault=false
    volumes:
      - /var/volumes/lamp/traefik/traefik.toml:/traefik.toml
      - /var/volumes/lamp/traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      external_net:
        ipv4_address: 1.1.1.1
      backend:

  web:
    build: web/.
    restart: always
    volumes:
      - /var/volumes/lamp/data:/export/data
      - /var/volumes/lamp/config:/export/config
    networks:
      backend:
    ports:
      - "8080:8080"
    depends_on:
      - db
    labels:
      - "traefik.enable=true"
      - "traefik.port=8080"
      - "traefik.frontend.rule=Host:hello.steamr.com"

  db:
    image: mariadb:10
    restart: always
    volumes:
      - /var/volumes/lamp/db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=xxxx
      - MYSQL_DATABASE=web
      - MYSQL_USER=web
      - MYSQL_PASSWORD=xxxx
    networks:
      - backend
        aliases:
          - mysql

networks:
  external_net:
    external: true

  backend:
    ipam:
      config:
        - subnet: 192.168.246.0/24

To bring this LAMP service up, run:

# ls
docker-compose.yml

##  the '-d' is for detached mode
# docker-compose up -d

docker-compose will then read the configuration file and set up the Docker networks and bring up the Docker containers with the required configs (image, volumes, network settings, environment variables, etc.) as defined in the file. Networking between different containers within the same service are done using dynamic linking automatically (rather than static links with hosts files) since individual containers can be recreated without bringing the entire stack down. The stack can be brought down using

# docker-compose down

If any changes are made to the docker-compose.yml file later on, the stack can be brought up to date by running docker-compose up again:

# docker-compose up -d

Networking

Some notes on networking in docker-compose:

Ports

Expose ports. Either specify both ports (HOST:CONTAINER), or just the container port (a random host port will be chosen).

  • Ports listed in docker-compose.yml will be accessible among all containers within this service.
  • These ports will be exposed to the host machine to a random port or a given port.

For example, we can start a MySQL server on port 3306. Any other container in this stack will be able to reach mysql:3306 (container-name:port). The DNS name is resolved by Docker's internal DNS service. The host machine is also able to reach the container, though its port will be randomized.

mysql:
  image: mysql:5.7
  ports:
    - "3306"

If you need MySQL exposed to the host on a specific port, such as 3306, pass in the HOST:CONTAINER port numbers. Since this binds to a specific port, be mindful of any existing ports in use either as containers or services running on the host.

mysql:
  image: mysql:5.7
  ports:
    - "3306:3306"

If the host machine has no firewall restricting access to port 3306, other machines that can reach the host machine will be able to talk on port 3306 to reach the MySQL service. If your host machine has multiple addresses, you may also bind to a specific address:

mysql:
  image: mysql:5.7
  ports:
    - "10.1.2.3:3306:3306"

Expose

Exposed ports will only be accessible to other containers within this stack. Like above, other containers can reach the MySQL server at mysql:3306 (container-name:port). Since this port is not exposed and bound to the host machine's network address, you may have multiple containers exposing the same ports without running into conflicts.

mysql:
  image: mysql:5.7
  expose:
    - "3306"

Networks

A service may connect to one or more networks. For example: 

services:
  nodered:
    image: nodered/node-red-docker:latest
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik
    networks:
      - traefik
      - mqtt-net
    expose:
      - "1880"
    restart: always

  broker:
    image: eclipse-mosquitto:1.6
    networks:
      - mqtt-net
    expose:
      - "1883"
    ports:
      - "10.1.2.200:1883:1883/tcp"
    restart: always

networks:
  mqtt-net:
  traefik:
    external:
      name: traefik

Each of the networks listed under nodered and broker must be defined in the network section in the docker-compose.yml file. Typically, you would want to associate a network with an actual docker network so that there is connectivity between other container stacks. In the example above, the mqtt-net network is private to only this container stack (and we don't really care what IPs they get, so long as they see each other) while the traefik network is connected to some external docker network which traefik (in another container stack) is connected to.

For more complex use cases, networks can be created with a different driver such as macvlan which allows for more advanced networking. For example, the following network configs were used for my Pi-Hole setup so that the container has access to a specific IP address on the network in order to allow DHCP to work. 

pihole:
    image: pihole/pihole
    restart: unless-stopped
    environment:
      - "TZ=MST7MDT,M3.2.0,M11.1.0"
      - ServerIP=10.1.0.8
    volumes:
      - '/var/volumes/pihole/etc-pihole/:/etc/pihole/'
      - '/var/volumes/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/'
    ports:
      - "10.1.0.8:53:53/tcp"
      - "10.1.0.8:53:53/udp"
      - "10.1.0.8:67:67/udp"
      - "10.1.0.8:80:80/tcp"
    cap_add:
      - NET_ADMIN
    #network_mode: "host"
    networks:
      pihole_network:
        ipv4_address: 10.1.0.8


networks:
  pihole_network:
    driver: macvlan
    driver_opts:
      parent: eth0
    ipam:
      config:
        - subnet: "10.1.0.0/22"
          gateway: "10.1.1.1"
          ip-range: "10.1.0.8/32"


Various Linux Notes
Linux Tools and Utilites
Package Management
Linux Distributions
Networking
Containers