GitLab

Installation

Docker based

Using docker-compose, we can quickly deploy GitLab with the following service definition. GitLab will be served on port 8888 with the registry served on port 5001. I then use Traefik (configured using container labels) to expose GitLab to the internet.

version: '3.3'

services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    container_name: gitlab
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://git.steamr.com'
        nginx['listen_port'] = '8888'
        nginx['listen_https'] = false
        nginx['proxy_set_headers'] = { 'X-Forwarded-Proto' => 'https', 'X-Forwarded-Ssl' => 'on' }
        registry_external_url 'https://registry.steamr.com'
        registry['enable'] = true
        gitlab_rails['registry_enabled'] = true
        registry_nginx['listen_port'] = 5001
        registry_nginx['listen_https'] = false
        registry_nginx['proxy_set_headers'] = { 'X-Forwarded-Proto' => 'https', 'X-Forwarded-Ssl' => 'on' }
    volumes:
      - /volumes/gitlab/config:/etc/gitlab
      - /volumes/gitlab/logs:/var/log/gitlab
      - /volumes/gitlab/data:/var/opt/gitlab
    restart: always
    expose:
      - "8888"
      - "5001"

With Kubernetes

GitLab provides a set of helm charts which can be used to deploy GitLab. Configuration is done with the ConfigMaps.

# helm repo add gitlab https://charts.gitlab.io/
# helm install gitlab/gitlab --name=gitlab --namespace=default --set global.operator.enabled=true --set global.operator.bootstrap=true -f gitlab-config.yml  --debug
# kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo

The unicorn/workhorse pod used to drive the GitLab web application gets its configs from the gitlab-unicorn ConfigMap.

Upgrading

When upgrading, review the change logs to determine the appropriate upgrade path. The upgrade paths that you should take can be determined using this tool: https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/

In general, your upgrade path needs to go through all the major revisions. For example, starting from 13.8.1 and upgrading to 14.3.0, you need to hop through to 13.8.Z, 13.9.Z, ... 13.12.Z, 14.0.Z, 14.1.Z, ..., 14.3.Z, where Z is the highest release number in that version. Use the upgrade path tool above to determine which versions are not required.

Upgrades should be scheduled periodically to avoid falling too far back as it may complicate the upgrade process and expose you to security vulnerabilities.

CI/CD Setup

GitLab has a built-in CI/CD pipeline feature that can automate certain tasks when code is committed into a repository. Some examples of CI/CD pipelines include compiling/building releases, running unit tests, or triggering certain web hooks when code is committed.

CI/CD pipelines can run in various ways. Most common is to use a Docker container to run CI/CD code in.

Docker based

I run the GitLab runner on a separate docker host from the GitLab instance. This is because GitLab runners will run with privileged mode and at worst pollutes the production docker environment with heaps of temporary build containers, and at worst can potentially interfere with the production GitLab environment.

Navigate to the settings page of a GitLab repository. Under CI/CD, go to 'Runners' and find the 'set up a specific runner manually' section. This section should contain the runner registration token.

To start the runner, I use the following docker command.

docker run --rm \
     -v /root/config:/etc/gitlab-runner \
     -v /var/run/docker.sock:/var/run/docker.sock \
     gitlab/gitlab-runner:latest \
     register -n --url https://git.steamr.com/ \
        --registration-token z276ujRTK69qD8xb-aNH \
        --executor docker \
        --docker-image "docker:stable"

Review the configuration that was generated and modify it as required. What I have is the following.

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800
  
[[runners]]
  name = "docker runner"
  url = "https://git.steamr.com/"
  token = "********"
  executor = "docker"
  [runners.docker]
    tls_cert_path = ""
    tls_verify = false
    image = "docker:stable"
    # alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

To start the runner, use the following command

docker run -d --name gitlab-runner --restart always \
     -v /root/config:/etc/gitlab-runner \
     -v /var/run/docker.sock:/var/run/docker.sock \
     gitlab/gitlab-runner:latest

Kubernetes based

To install a GitLab runner with Helm, install the GitLab repo and then the helm chart. You might want to use an existing values.yaml file from https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml as a reference.

# helm repo add gitlab https://charts.gitlab.io
# helm install \
        --namespace gitlab \
        --name gitlab-runner \
        -f values.yaml \
        --set gitlabUrl=https://git.caas.ucalgary.ca,runnerRegistrationToken=xxxxxxxxxxx \
        gitlab/gitlab-runner

Troubleshooting

Failed to create listener - address family not supported

Runner container crashes with a fatal error message:

FATAL: Failed to create listener for metrics server  builds=0 error=listen tcp [::]:9252: socket: address family not supported by protocol

To fix, set in values.yaml:

envVars:
  - name: LISTEN_ADDRESS
    value: :9252

Docker-based CI/CD runner fails - docker no such host

My CI/CD pipeline fails with the following error:

ERROR: error during connect: Get http://docker:2375/v1.40/info: dial tcp: lookup docker on 10.1.2.200:53: no such host

This is caused because the runner is trying to reach the docker engine via the 'docker' DNS name rather than with the path based socket. To fix this, ensure that the docker runner has /var/run/docker.sock mounted as a volume and that the runner config has the following two lines:

privileged = true
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]

Remove/restart the runner and restart the pipeline.

Tasks

Registry Cleanup

The docker registry does not remove deleted images. Like a bad memory leak, the registry will grow as old images are still stored even when dereferenced and deleted.

Use the docker-distribution-pruner utility which can be obtained from their project CI / CD page:

• https://gitlab.com/gitlab-org/docker-distribution-pruner/-/jobs

On the GitLab instance, run the following:

## This will soft-delete the files to /var/opt/gitlab/gitlab-rails/shared/registry/docker-backup
# EXPERIMENTAL=true ./docker-distribution-pruner -config=/var/opt/gitlab/registry/config.yml -delete
INFO[0000] Walking REPOSITORIES...
INFO[0000] REPOSITORIES DIR: repositories
INFO[0000] Walking BLOBS...
...
INFO[0001] BLOBS INFO: Objects/Unused: 296 / 75 Data/Unused: 3.0 GB / 398 MB
WARN[0001] DELETEABLE INFO: 115 links, 75 blobs, 0 other, 398 MB

## Remove the files
# rm -rf /var/opt/gitlab/gitlab-rails/shared/registry/docker-backup

Backups

GitLab using the Helm Chart can be backed up by running backup-utility inside the GitLab runner container. Backups that are generated will be placed at /srv/gitlab/tmp/backup as defined by the backup.path parameter in /srv/gitlab/config/gitlab.yml.

Omnibus GitLab can be backed up by scheduling a task that runs /opt/gitlab/bin/gitlab-rake gitlab:backup:create. The destination is configured based on the gitlab_rails['backup_path'] value in /etc/gitlab/gitlab.rb. Specific components can be skipped by passing in the SKIP value. For example:

# /opt/gitlab/bin/gitlab-rake gitlab:backup:create SKIP=registry,pages,artifacts,builds  DIRECTORY=daily

See Also:

• https://docs.gitlab.com/ee/raketasks/backup_restore.html
• https://docs.gitlab.com/omnibus/settings/backups.html
• https://docs.gitlab.com/charts/backup-restore/restore.html

To restore, run gitlab-rake gitlab:backup:restore BACKUP=1571687885_2019_10_21_12.3.5-ee, to target a backup file in the backup directory named 1571687885_2019_10_21_12.3.5-ee_gitlab_backup.tar.

git@gitlab-task-runner-6c64cd44f8-wvjb4:/srv/gitlab/tmp/backups$ gitlab-rake gitlab:backup:restore BACKUP=1571687885_2019_10_21_12.3.5-ee
WARNING: This version of GitLab depends on gitlab-shell 10.0.0, but you're running Unknown. Please update gitlab-shell.
Unpacking backup ...