Kubernetes

This article is still in progress as I try to understand everything about Kubernetes. It is in no way complete.

Kubernetes is an open source platform for managing containerized services first created by Google.

The name originates from Greek for helmsman or pilot (Hence, their ship's wheel as their logo). The 8 letters in the middle is commonly replaced with '8' with the entire name stylized as 'k8s'.

Current version of Kubernetes is 1.13.

Installation

There are two main options when building a Kubernetes cluster:

1. Build it manually
2. Build it with automated tools such as Terraform and Ansible.

Automated

Automatic provisioning can be done using deployment tools such as:

• Terraform
• Kubicorn

Manual

• https://www.linuxtechi.com/install-kubernetes-1-7-centos7-rhel7/

Master Node

# firewall-cmd --permanent --add-port=6443/tcp
# firewall-cmd --permanent --add-port=2379-2380/tcp
# firewall-cmd --permanent --add-port=10250/tcp
# firewall-cmd --permanent --add-port=10251/tcp
# firewall-cmd --permanent --add-port=10252/tcp
# firewall-cmd --permanent --add-port=10255/tcp
# firewall-cmd --reload
## Kubernetes does not handle memory evictions, disable swap. remember to remove it from /etc/fstab too.
# swapoff -a
# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
# yum -y install kubeadm docker
# systemctl enable docker kubelet
# systemctl start docker kubelet
# kubeadm init

# export kubever=$(kubectl version | base64 | tr -d '\n')
# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"

Once everything finishes running, it may take a few more minutes before the node becomes ready as the WeaveWorks network containers get pulled and started.

Dashboard

To get the Web UI Dashboard working, see:

• https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#deploying-the-dashboard-ui
• https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html

It boils down to:

# kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
# kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
# kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
# kubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml

## Create a service account with permission:
# cat <<EOF > eks-admin-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-admin
  namespace: kube-system
EOF
# kubectl apply -f eks-admin-service-account.yaml
# cat <<EOF > eks-admin-cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: eks-admin
  namespace: kube-system
EOF
# kubectl apply -f eks-admin-cluster-role-binding.yaml

## Get the access token
# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret

Kubernetes (worker) Nodes

# firewall-cmd --permanent --add-port=10250/tcp
# firewall-cmd --permanent --add-port=10255/tcp
# firewall-cmd --permanent --add-port=30000-32767/tcp
# firewall-cmd --permanent --add-port=6783/tcp
# firewall-cmd  --reload
## Kubernetes does not handle memory evictions, disable swap. remember to remove it from /etc/fstab too.
# swapoff -a
# modprobe br_netfilter
# for i in ip_vs ip_vs_sh ip_vs_rr ip_vs_wrr ; do modprobe $i ; done

# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

# yum -y install kubeadm docker
# systemctl enable docker kubelet
# systemctl start docker kubelet

To join the worker to the cluster, obtain the join token as well as the discovery token CA cert hash. On the master node, run:

# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
4c6ee8dd02afae0f9231c1002a4fca671622addc7ff4b522a70d68ed0b525445
# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
a0mjyg.eiqvzbpwf2dgn9z3   <invalid>   2018-11-29T11:57:02-07:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token

Based on the values above, on the worker node, run:

# kubeadm join --token a0mjyg.eiqvzbpwf2dgn9z3 --discovery-token-ca-cert-hash sha256:4c6ee8dd02afae0f9231c1002a4fca671622addc7ff4b522a70d68ed0b525445 10.1.3.252:6443

Alternatively, you could generate a new token on the master node:

# kubeadm token create --print-join-command
kubeadm join 10.1.3.252:6443 --token so4o6b.fyzvgwpgmpoq6mdm --discovery-token-ca-cert-hash sha256:4c6ee8dd02afae0f9231c1002a4fca671622addc7ff4b522a70d68ed0b525445

Quick Start

To get an application running as a service:

• Create a deployment which defines how many replicas should exist, the container to use, and exposed ports.
  • This will create a Pod. Each replica creates an additional Pod (that ideally resides on different nodes).
  • kubectl get deployments to see deployments
  • kubectl get pods to see pods
• Create a service which defines the name of the service and the port the service is exposed on
  • kubectl get service

Concepts

See the Kubernetes documentation at https://kubernetes.io/docs/concepts/

In summary:

• A Deployment defines the desired state for pods and ReplicaSets.
• A ReplicaSets creates or destroys pods depending on scaling and the number of pods that are running.
• A Pod is a collection of container(s) all residing on one node.
• A Service is an abstraction that defines the logical set of Pods. The Pods could be turned off or migrated without affecting the overall Service.
• A Kubernetes Master is responsible for maintaining the state of the Kubernetes cluster.
• A Kubernetes (Worker) Node is responsible for running the actual applications managed by Kubernetes.

Each concept will be covered in more detail below.

Pods

A pod is:

• A collection of application containers
• Guaranteed to be deployed on the same Kubernetes cluster node
• Shares the same cgroup, IP address, hostname (hence, all containers in a pod runs in the same execution environment)

A pod should provide one individual component of an application that can:

• Be scaled independently of all other components in the application (eg. Database with respect to the frontend web server)
• Work even if placed (ie. orchestrated) on a different machine

In general, the right question to ask yourself when designing Pods is, “Will these containers work correctly if they land on different machines?” If the answer is “no,” a Pod is the correct grouping for the containers. If the answer is “yes,” multiple Pods is probably the correct solution. In the example at the beginning of this chapter, the two containers interact via a local filesystem. It would be impossible for them to operate correctly if the containers were scheduled on different machines.

—Thinking with Pods, Kubernetes: Up and Running

Pods are assigned a unique Pod IP address within the cluster. All containers inside a pod can reference each other via localhost. Containers outside a pod can only reference other containers in other containers using the Pod IP address or via a Service.

Pods are defined in text file as a manifest. The Kubernetes API server processes the manifest, then stores it in persistent storage (etcd). A scheduler then finds pods that need to be scheduled and deploys the pods on the appropriate resource that satisfies any constraints defined in the manifest.

A Pod can be created by invoking a command: kubectl run kuard --image=registry/something/something:tag. However, using a manifest (aka. an object definition in yaml) to define the Pod is more manageable. A manifest looks something like this:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Pod
metadata:
  name: kuard
spec:
  containers:
    - image: gcr.io/kuar-demo/kuard-amd64:1
      name: kuard
      ports:
        - containerPort: 8080
          name: http
          protocol: TCP
      env:
        - value: something
      resources:
        requests:
          cpu: 100m
          cpu: 100Mi

A few things to note from the definition above (and this applies to all manifests)

1. Kind: Specifies the kind of Kubernetes resource this manifest defines
2. Metadata: Helpful data to uniquely identify the object. Eg. name, UID, optionally namespace
3. Spec: Object data; for Pods, it should be an array of containers (including container image, name, ports, environment variables, resources, etc).

For more information on the spec, see https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md.

• Create using kubectl apply -f pod-manifest.yml
• See it using kubectl get pods
• Detailed information about it using kubectl describe pods pod-name
• Delete it using kubectl delete pods/pod-name or kubectl delete -f pod-manifest.yml

Pods that are set for deletion will cease to have new requests sent to it. After a 30 second termination grace period, the pods are then terminated. This extra time allows for the pod to reliably finish active requests.

Controllers

A controller is a reconciliation loop that drives the actual cluster state to the desired cluster state.

A few types of controllers are controlled by the following 'set' objects.

ReplicaSet

See: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/

ReplicaSets defines the number of pod replicas that are running at a given time and is enforced by the Replication Controller. Typically, ReplicaSets are used by Deployments as a mechanism to orchestrate pod creation, deletion, and updates.

DaemonSet

See: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

DaemonSet defines which Pod will run on some or all Kubernetes Nodes. As nodes are added or removed, Pods will be created or destroyed with it. Some use cases requring DaemonSets include storage services (glusterd, ceph), monitoring, log collection, etc., on each node.

Others

Stateful Sets: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ Job Controller: Runs a Pod as a job. TBD.

Deployments

See: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

Deployments defines the desired state of Pods or ReplicaSets which are enforced by the Deployment Controller. A deployment can be versioned so that Kubernetes can let you rollout a new version with the ability to pause or even rollback the changes at a later time.

The manifest's spec should contain a template that contains the information about a new Pod.

Example Deployment manifest:

No code provided.

• Create using kubectl apply -f frontend-deployment.yml
• See it using kubectl get deployments
• Pods that this deployment creates can be seen using the label selector.
  • This example manifest has 2 labels applied: app, and tier
  • See it using kubectl get pods -l app=guestbook -l tier=frontend

You can change the scale of a deployment:

[root@kube guestbook]# kubectl get deployment frontend
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
frontend   3         3         3            3           44m
[root@kube guestbook]# kubectl scale deployment frontend --replicas=5
deployment.extensions/frontend scaled
[root@kube guestbook]# kubectl get deployment frontend
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
frontend   5         5         5            5           45m
[root@kube guestbook]# kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
frontend-654c699bc8-5ngzj       1/1     Running   0          45m
frontend-654c699bc8-77b58       1/1     Running   0          45m
frontend-654c699bc8-ll7cr       1/1     Running   0          22s
frontend-654c699bc8-pqzvp       1/1     Running   0          22s
frontend-654c699bc8-sq682       1/1     Running   0          45m

Service

A Service provides an abstraction between the user and the underlying Pod or Pods providing the service. Since Pods are ephemeral, their IP addresses may change as they get created and destroyed and individual pods may go down; a Service provides applications with a name and load balancing and routing to maintain the service's availability.

A Service by default provides a single IP address for the set of pods (known as a ClusterIP) and are only accessible within the cluster. This can be changed so that a Service provides a load balanced port (LoadBalancer) or a port which is exposed on the node (NodePort).

An example manifest:

apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: guestbook
    tier: frontend
spec:
  # comment or delete the following line if you want to use a LoadBalancer
  type: NodePort 
  # if your cluster supports it, uncomment the following to automatically create
  # an external load-balanced IP for the frontend service.
  # type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: guestbook
    tier: frontend

• Apply the Service using kubectl apply -f frontend-service.yaml
• See Services using kubectl get services

If you use a NodePort and the service looks like this:

[root@kube guestbook]# kubectl get service frontend
NAME       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
frontend   NodePort   10.97.195.242   <none>        80:32764/TCP   10m

You can get to your service by accessing the Node on port 32764.

Namespaces

A namespace organizes objects in the cluster. It is analogous to OU containers in Active Directory, folders in a filesystem, or classes in object oriented languages. By default, new objects are placed in the 'default' namespace.

Objects cannot see other objects in different namespaces (this applies even to the default namespace). Secrets in a different namespace will not be visible.

Kubernetes users can be assigned permissions to specific namespaces which can be used to limit a user's access on a shared multi-tenant cluster.

Contexts

Contexts are like a profile. A context can have different default namespace, or user credentials to manage different clusters.

Change the current context using kubectl config use-context my-context

Config File

Located in ~/.kube/config. This file contains credentials to authenticate to the cluster.

It contains the default namespace and context values.

Kubernetes API

The Kubernetes API is a RESTful API, providing access to the Kubernetes backend.

Objects in the Kubernetes API are represented as JSON or Yaml files. Files can be used to create, update, or delete objects from the server.

Description	Command
Create/Update	# kubectl apply -f obj.yaml
Edit	# kubectl edit <resource-name> <object-name>
Delete	# kubectl delete -f obj.yaml ## or # kubectl delete <resource-name> <object-name>

All objects can be annotated or given a label.

Description	Command
Label pod 'bar' color=red	# kubectl label pods bar color=red ## pass --overwrite if it already exists.
Remove label color from pod 'bar'	# kubectl label pods bar -color

Kubernetes Master Node

A Kubernetes master node contains containers that provide the API server, scheduler, etc. that manages the cluster.

The master node should have the following components:

• controller-manager: Responsible for running controllers that regulate behavior int he cluster. Eg. ensure replicas for a service are available and healthy.
• scheduler: Places pods into different nodes in the cluster
• etcd: storage for cluster; stores API objects.

All components deployed by Kubernetes run under the kube-system namespace.

# kubectl describe nodes kube
...
Non-terminated Pods:         (8 in total)
  Namespace                  Name                            CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                            ------------  ----------  ---------------  -------------
  kube-system                coredns-576cbf47c7-6mphw        100m (2%)     0 (0%)      70Mi (0%)        170Mi (2%)
  kube-system                coredns-576cbf47c7-75n6g        100m (2%)     0 (0%)      70Mi (0%)        170Mi (2%)
  kube-system                etcd-kube                       0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-apiserver-kube             250m (6%)     0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-controller-manager-kube    200m (5%)     0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-proxy-cmdsn                0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-scheduler-kube             100m (2%)     0 (0%)      0 (0%)           0 (0%)
  kube-system                weave-net-swwgs                 20m (0%)      0 (0%)      0 (0%)           0 (0%)
...

The Kubernetes proxy is responsible for routing network traffic to services in the kubernetes cluster. (Question: Does it do the load balancing?). A proxy exists on every node.

# kubectl get daemonsets --namespace=kube-system
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-proxy   1         1         1       1            1           <none>          26h
weave-net    1         1         1       1            1           <none>          28m

Question: What is a DaemonSet?

Kubernetes also runs a DNS server that provides naming and discovery for services in the cluster.

# kubectl get deployments --namespace=kube-system
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
coredns   2         2         2            2           26h

# kubectl get services --namespace=kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   26h

The DNS service for the cluster runs on 10.96.0.10. If you log into a container in the cluster, this server will be used as the primary DNS server.

Kubernetes Dashboard UI can be installed. Like the DNS service, it is both a deployment and a service:

# kubectl get deployments --namespace=kube-system kubernetes-dashboard
NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kubernetes-dashboard   1         1         1            1           2m26s

[root@kube ~]# kubectl get services --namespace=kube-system kubernetes-dashboard
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   ClusterIP   10.108.158.128   <none>        443/TCP         2m36s

Run kubectl proxy to proxy the server on localhost:8001 and then access it in a web browser at http://localhost:8001/ui. If this is on a remote server, create a SSH tunnel.

Kubernetes Node

Also known as a Worker or Minion node which has a container runtime such as Docker.

The Kubelet is a daemon on each node that will start/stop/maintain application containers as directed by the Kubernetes Master (the orchestrator/scheduler/control plane).

The scheduler does this by checking a node's taint and will not schedule pods on nodes that contain things like node-role.kubernetes.io/master:NoSchedule. Attempting to do so will result in a FailedScheduling status and a message of 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. when looking at pod events using kubectl describe pods pod-name.

Commands

The kubectl command line tool is the official kubernetes client for interacting with the Kubernetes API.

Description	Command
Get all nodes	# kubectl get nodes
Get all pods	# kubectl get pods --all-namespaces
Get information about a node	# kubectl describe nodes
See components in the cluster	# kubectl get componentstatuses

Tips: When using kubectl get, pass

• --no-headers to remove headers for easier parsing
• -o json|yaml to format output in json/yaml.

Description	Command
Create a pod	# kubectl run kuard --image=gcr.io/kuar-demo/kuard-amd64:1 # kubectl apply -f kuard-pod.yaml
Listing pods	# kubectl get pods ## Filter by label with -l, can supply multiple of these. # kubectl get pods -l label=something
Delete pod	# kubectl delete deployments/kuard # kubectl delete -f kuard-pod.yaml
Pod Details	# kubectl describe pods kuard
Pod Logs	# kubectl logs kuard
Enter a container	# kubectl exec kuard cmd # kubectl exec -it kuard sh
Copy to/from container	# kubectl cp podname:/src ./dst # kubectl cp ./src podname:/dst

A pod manifest looks something like this:

apiVersion: v1
kind: Pod
metadata:
  name: kuard
spec:
  containers:
    - image: gcr.io/kuar-demo/kuard-amd64:1
      name: kuard
      ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Questions

• What is involved in setting up a cluster on multiple VMs?
• What is the Kubernetes API?
• What is this persistent storage (etcd)?
• What is the WeaveWorks network and how does it work?

See Also

• Command cheat sheet
• Kubernetes: Up & Running
• CouchDB failover demo: https://blog.couchbase.com/databases-on-kubernetes/
• A Kubernetes Guide
• https://kubernetes.io/docs/tutorials/stateless-application/guestbook/ Simple overview on deploying a guestbook application

• MySQL on Kubernetes https://www.youtube.com/watch?v=J7h0F34iBx0 Not exactly beginner friendly. Goes over the MySQLOperator and Vitess