GitLab Migration and Upgrade

I’ve worked with my mate Donovan Winter (we are called Thomson and Thompson!) on a very interesting project for one of our customer that’s worth sharing. They had an old version of GitLab (release 13.3.4) running in a pod in OpenShift and wanted to migrate it into a Virtual Machine under Rocky Linux 8.

They then wanted to upgrade it to the latest GitLab release (release 15.7.3). I’ll share here all the steps required for doing such migration and upgrade.

Setting up the new GitLab Server

The first step is to prepare a new Virtual Machine and install the prerequisite packages and services for GitLab:

$ sudo dnf install -y curl openssh-server perl checkpolicy policycoreutils-python-utils python3-audit python3-libsemanage python3-setools python3-policycoreutils policycoreutils-python-utils

$ sudo systemctl enable sshd
$ sudo systemctl start sshd

$ sudo firewall-cmd --permanent --add-service=http
$ sudo firewall-cmd --permanent --add-service=https
$ sudo systemctl reload firewalld

Then install GitLab 13.3.4 on this new server as this is the same release as the one in OpenShift:

$ curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh| sudo bash

$ sudo dnf install gitlab-ce-13.3.4-ce.0.el8

The curl command above adds the GitLab repository in our system so we can then install the GitLab package in release 13.3.4 (in our case). Note also that there are 2 flavours of GitLab: ce (Community Edition) and ee (Enterprise Edition). We installed the ce one because this is the one used in OpenShift and it has to match in order to be able to backup/restore data between these 2 instances.

Then, set the initial GitLab configuration by editing the file /etc/gitlab/gitlab.rb and removing the # in front of the line # gitlab_rails[‘initial_root_password’] = “password”

We also changed in this file the URL of the GitLab Server on which to connect for the graphical interface: external_url ‘http://<url_of_this_server>’

Each time this GitLab configuration file is changed, you’ll need to reconfigure GitLab to take those modifications into account with the following command:

$ sudo gitlab-ctl reconfigure

In our case we had the following errors after running the reconfigure command:

Running handlers:
There was an error running gitlab-ctl reconfigure:
execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] (postgresql::enable line 75) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8 ----
STDOUT: The files belonging to this database system will be owned by user "gitlab-psql".
This user must also own the server process.
STDERR: initdb: invalid locale settings; check LANG and LC_* environment variables
---- End output of /opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8 ----
Ran /opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8 returned 1

This can be resolved as follows:

$ export LC_ALL="en_US.UTF-8"
$ export LC_CTYPE="en_US.UTF-8"

$ sudo gitlab-ctl reconfigure

And then the reconfigure was successful.

Backup of GitLab in OpenShift

Connect to the OpenShift cluster, backup GitLab and transfer the backup file to the new GitLab server.

$ oc login --server=https://<cluster-server-name> -u admin
$ oc project gitlab-prod

$ oc get pods
NAME                READY   STATUS    RESTARTS   AGE
gitlab-ce-9-p5xzx   1/1     Running   0          9d
redis-2-jwxzl       1/1     Running   0          1y

$ oc describe pod/gitlab-ce-9-p5xzx

In an OpenShift cluster, you first login to it and then enter into a project (which is the equivalent of a namespace in Vanilla Kubernetes).

There are only 2 pods in this project, one that contain the GitLab server and another one with the Redis database. There is no GitLab operator pod that would have helped creating a backup of this instance.

Using describe on this GitLab pod will show all the environment variables that we may need in our new GitLab server. I’ll come back to this later on.

We’ll then have to connect to the GitLab pod and do the backup as shown below:

$ oc rsh gitlab-ce-9-p5xzx
# su git
git@gitlab-ce-9-p5xzx:~/gitlab$ /home/git/gitlab/bin/rake gitlab:backup:create

This GitLab pod didn’t contain any of the usual GitLab binaries in /usr/bin so we can only rely on the rake tool that is part of the core of GitLab in order to do a backup. The backup file is then stored into the folder /home/git/data/backups of the pod as configured in its configuration file /etc/gitlab/gitlab.rb

We then just need to transfer this backup file using the OpenShift command below and transfer it to the new GitLab server:

$ oc rsync gitlab-ce-9-p5xzx:/home/git/data/backups/1673909524_2023_01_31_13.3.4_gitlab_backup.tar ./

$ scp <source> <destination>

Restore data in the new GitLab Server

First, set the backup path into the file /etc/gitlab/gitlab.rb by uncommenting and editing the following line : gitlab_rails[‘backup_path’] = “/data/gitlab/backups”

Create this new backup repository and make git the owner :

$ sudo mkdir -p /data/gitlab/backups && sudo chown -R git:git /data/gitlab

Copy the backup file to this new repository and set the owner of this file to git:

$ sudo cp 1673909524_2023_01_31_13.3.4_gitlab_backup.tar /data/gitlab/backups/

$ sudo chown git:git /data/gitlab/backups/1673909524_2023_01_31_13.3.4_gitlab_backup.tar

Stop only the GitLab processes that are connected to the database:

$ sudo gitlab-ctl stop unicorn 
$ sudo gitlab-ctl stop puma 
$ sudo gitlab-ctl stop sidekiq

Reconfigure to apply changes in the gitlab.rb file (the backup path set above)

$ sudo gitlab-ctl reconfigure

As the backup file has been copied into the backup folder of the new GitLab Server, we just have to restore it and restart everything:

$ sudo gitlab-backup restore BACKUP=1673909524_2023_01_31_13.3.4

$ sudo gitlab-ctl reconfigure 
$ sudo gitlab-ctl restart
$ sudo gitlab-rake gitlab:check SANITIZE=true

At this stage we have successfully migrated GitLab 13.3.4 from OpenShift to a new VM. Let’s now continue with the upgrade process to the latest release.

GitLab Upgrade

GitLab provides a great tool (thank you to my mate Donovan Winter – one of our rising star in DevOps and our GitLab guru – for finding this out!) to see at a glance the upgrade path to follow by using the following link https://gitlab-com.gitlab.io/support/toolbox/upgrade-path

Below is the path to follow:

[`13.1.11`](https://docs.gitlab.com/ee/update/#1310)

-> [`13.8.8`](https://docs.gitlab.com/ee/update/#1388)

-> [`13.12.15`](https://docs.gitlab.com/ee/update/#13120)

-> [`14.0.12`](https://docs.gitlab.com/ee/update/#1400)

-> [`14.3.6`](https://docs.gitlab.com/ee/update/#1430)

-> [`14.9.5`](https://docs.gitlab.com/ee/update/#1490)

-> [`14.10.Z`](https://docs.gitlab.com/ee/update/#14100)

-> [`15.0.Z`](https://docs.gitlab.com/ee/update/#1500)

-> [`15.1.Z`](https://docs.gitlab.com/ee/update/#1510) (for GitLab instances with multiple web nodes)

-> [`15.4.0`](https://docs.gitlab.com/ee/update/#1540)

-> [latest `15.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases)

Below are all the commands of this upgrade path:

# Upgrade to 13.8.8
$ sudo dnf install -y gitlab-ce-13.8.8

# Upgrade to 13.12.15
$ sudo dnf install -y gitlab-ce-13.12.15

# Upgrade to 14.0.12
$ sudo dnf install -y gitlab-ce-14.0.12

# Upgrade to 14.3.6
$ sudo dnf install -y gitlab-ce-14.3.6
$ sudo gitlab-ctl restart redis
$ sudo gitlab-ctl restart postgresql
$ sudo gitlab-rake db:migrate

# Upgrade to 14.9.5
$ sudo dnf install -y gitlab-ce-14.9.5

# Upgrade to 14.10.5
$ sudo dnf install -y gitlab-ce-14.10.5

# Upgrade to 15.0.5
$ sudo dnf install -y gitlab-ce-15.0.5

# Upgrade to 15.4.6
$ sudo dnf install -y gitlab-ce-15.4.6
$ sudo gitlab-ctl restart redis

# Upgrade to 15.7.3
$ sudo dnf install -y gitlab-ce-15.7.3
$ sudo gitlab-ctl restart redis

When Redis or Postgresql needs to be restarted, it is mentioned at the end of output of the release installation. For some release, this is required because the running service version is different than the new one.

The new GitLab server is now upgraded to release 15.7.3

Final configuration

You may remember that at the very beginning we’ve looked at the environment variables of the GitLab pod. It is now time to transfer the required parameters to this new shiny GitLab server to complete our migration. In our case we only had to set back the parameters used for SMTP. For everything else, we kept the default settings of the new GitLab server. Below are the SMTP parameters of the file /etc/gitlab/gitlab.rb

gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'no-reply@<smtp_domain>'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.<smtp_domain>"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_domain'] = "smtp_domain"

As usual we reconfigure GitLab to take those parameters into account:

$ sudo gitlab-ctl reconfigure

To test that SMTP is working properly, you can do a test and send an email to yourself, if it works (and it did for us) you are all set!

$ sudo gitlab-rails console
> Notify.test_email('[email protected]', 'Benoit Test', 'Test ok :)').deliver_now

Migration and Upgrade done!

This is it! Those were all the steps required to migrate and upgrade a GitLab instance from a pod in OpenShift to a Virtual Machine.

I hope this will help you if you have to do the same operation. Note that this procedure is also valid if you want to migrate and upgrade from another server (or Virtual Machine) instead of a pod in a Kubernetes cluster or if you just want to upgrade a GitLab instance. Just follow the steps in the section that match your needs!

For a migration from another GitLab server, the backup is easily done with the command sudo gitlab-backup create on that instance and then you can follow this procedure for the restore (and upgrade if required).

Have fun with it!