GitLab
Registry Cleanup
The docker registry does not remove deleted images. Like a bad memory leak, the registry will grow as old images are still stored even when dereferenced and deleted.
Use the docker-distribution-pruner utility which can be obtained from their project CI / CD page:
On the GitLab instance, run the following:
## This will soft-delete the files to /var/opt/gitlab/gitlab-rails/shared/registry/docker-backup
# EXPERIMENTAL=true ./docker-distribution-pruner -config=/var/opt/gitlab/registry/config.yml -delete
INFO[0000] Walking REPOSITORIES...
INFO[0000] REPOSITORIES DIR: repositories
INFO[0000] Walking BLOBS...
...
INFO[0001] BLOBS INFO: Objects/Unused: 296 / 75 Data/Unused: 3.0 GB / 398 MB
WARN[0001] DELETEABLE INFO: 115 links, 75 blobs, 0 other, 398 MB
## Remove the files
# rm -rf /var/opt/gitlab/gitlab-rails/shared/registry/docker-backup
Docker Registry Runners
# docker-compose run --rm gitlab-runner register -n --url https://git.steamr.com/ --registration-token z276ujRTK69qD8xb-aNH --executor docker --docker-image "docker:stable"
Runtime platform arch=amd64 os=linux pid=7 revision=3afdaba6 version=11.5.0
Running in system-mode.
Registering runner... succeeded runner=z276ujRT
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
root@docker:/root/docker/running/gitlab# cat /var/volumes/gitlab/runner-config/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "3b70de49f826"
url = "https://git.steamr.com/"
token = "1dbe4535e5aaa49d6582d374a29c28"
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:stable"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
root@docker:/root/docker/running/gitlab# cat docker-compose.yml /root/docker/running/gitla
version: '3.3'
services:
gitlab:
#build:
# context: ./gogs
# dockerfile: Dockerfile
# letsencrypt['enable'] = false
image: gitlab/gitlab-ce:latest
container_name: gitlab
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'https://git.steamr.com'
nginx['listen_port'] = '80'
nginx['listen_https'] = false
nginx['proxy_set_headers'] = { 'X-Forwarded-Proto' => 'https', 'X-Forwarded-Ssl' => 'on' }
registry_external_url 'https://registry.steamr.com'
registry['enable'] = true
gitlab_rails['registry_enabled'] = true
registry_nginx['listen_port'] = 5001
registry_nginx['listen_https'] = false
registry_nginx['proxy_set_headers'] = { 'X-Forwarded-Proto' => 'https', 'X-Forwarded-Ssl' => 'on' }
volumes:
- /var/volumes/gitlab/config:/etc/gitlab
- /var/volumes/gitlab/logs:/var/log/gitlab
- /var/volumes/gitlab/data:/var/opt/gitlab
restart: always
expose:
- "80"
- "5001"
ports:
- "10.1.2.54:22:22"
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik"
- "traefik.web.port=80"
- "traefik.web.frontend.rule=Host:git.steamr.com"
- "traefik.registry.port=5001"
- "traefik.registry.frontend.rule=Host:registry.steamr.com"
networks:
- traefik
healthcheck:
disable: true
gitlab-runner:
image: gitlab/gitlab-runner:latest
container_name: gitlab-runner
restart: always
volumes:
- /var/volumes/gitlab/runner-config:/etc/gitlab-runner
- /var/run/docker.sock:/run/docker.sock
networks:
traefik:
external:
name: traefik
Kubernetes GitLab Runner
To install a GitLab runner with Helm, install the GitLab repo and then the helm chart. You might want to use an existing values.yaml file from https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/master/values.yaml as a reference.
# helm repo add gitlab https://charts.gitlab.io
# helm install \
--namespace gitlab \
--name gitlab-runner \
-f values.yaml \
--set gitlabUrl=https://git.caas.ucalgary.ca,runnerRegistrationToken=xxxxxxxxxxx \
gitlab/gitlab-runner
Kubernetes GitLab
# helm repo add gitlab https://charts.gitlab.io/
# helm install gitlab/gitlab --name=gitlab --namespace=default --set global.operator.enabled=true --set global.operator.bootstrap=true -f gitlab-config.yml --debug
# kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode ; echo
Configuration files are set inside ConfigMaps. The unicorn/workhorse pod used to drive the GitLab web application gets its configs from the gitlab-unicorn ConfigMap.
Troubleshooting
Runner container crashes with a fatal error message:
FATAL: Failed to create listener for metrics server builds=0 error=listen tcp [::]:9252: socket: address family not supported by protocol
To fix, set in values.yaml:
envVars:
- name: LISTEN_ADDRESS
value: :9252
Backups
GitLab using the Helm Chart can be backed up by running backup-utility inside the GitLab runner container. Backups that are generated will be placed at /srv/gitlab/tmp/backup as defined by the backup.path parameter in /srv/gitlab/config/gitlab.yml.
Omnibus GitLab can be backed up by scheduling a task that runs /opt/gitlab/bin/gitlab-rake gitlab:backup:create. The destination is configured based on the gitlab_rails['backup_path'] value in /etc/gitlab/gitlab.rb. Specific components can be skipped by passing in the SKIP value. For example:
# /opt/gitlab/bin/gitlab-rake gitlab:backup:create SKIP=registry,pages,artifacts,builds DIRECTORY=daily
See Also:
- https://docs.gitlab.com/ee/raketasks/backup_restore.html
- https://docs.gitlab.com/omnibus/settings/backups.html
- https://docs.gitlab.com/charts/backup-restore/restore.html
To restore, run gitlab-rake gitlab:backup:restore BACKUP=1571687885_2019_10_21_12.3.5-ee, to target a backup file in the backup directory named 1571687885_2019_10_21_12.3.5-ee_gitlab_backup.tar.
git@gitlab-task-runner-6c64cd44f8-wvjb4:/srv/gitlab/tmp/backups$ gitlab-rake gitlab:backup:restore BACKUP=1571687885_2019_10_21_12.3.5-ee
WARNING: This version of GitLab depends on gitlab-shell 10.0.0, but you're running Unknown. Please update gitlab-shell.
Unpacking backup ...
| ||||||