Minor PostgreSQL version upgrades in a Patroni cluster

Last week PostgreSQL 14.1, and all other minor versions for the supported version of PostgreSQL have been released. As usual, it is highly recommended to update to the latest minor release as soon as possible. Minor version updates of PostgreSQL are just a matter of installing the new binaries, shutting down the old cluster and starting it up again using the new binaries, that’s it. If you are using packages the procedure is to shutdown the cluster, install the new packages and start it up again. Nothing really complicated in either case and really easy to do. What we’ll look into in this post is, how you can do minor version PostgreSQL updates if you run Patroni on top of PostgreSQL for implementing a high available solution.

My current Patroni configuration looks like this:

postgres@patroni1:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

There is a primary/leader instance which feeds to replicas, all of them running PostgreSQL 14.0:

postgres@patroni1:/home/postgres/ [PG1] psql -c "select version()";
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.0 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

What do we need to do to update this configuration to PostgreSQL 14.1? I’ve installed from source code, so the first step is to install the new version of PostgreSQL. This is already done in may case:

postgres@patroni1:/home/postgres/ [PG1] ls -l /u01/app/postgres/product/14/
total 8
drwxr-xr-x 6 postgres postgres 4096 Nov 10 20:43 db_0
drwxr-xr-x 6 postgres postgres 4096 Nov 11 15:35 db_1

“db_0” is PostgreSQL 14.0 and “db_1” is PostgreSQL 14.1. This is our standard, in your case this of course very probably looks different. Having both versions installed on all nodes already, what is the next step? First you need think about which node you want to upgrade first. We usually go for one of the replicas and in the case of PostgreSQL 14.1 this is even recommended as one patch (“Fix physical replication for cases where the primary crashes after shipping a WAL segment that ends with a partial WAL record (Álvaro Herrera)”) should be applied on the replica first.

Let’s start with the third node (patroni3/192.168.100.172). First we need to stop Patroni on that host:

postgres@patroni3:/home/postgres/ [PG1] sudo systemctl stop patroni

Checking the Patroni status afterwards, we can see that the node is now gone:

postgres@patroni3:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

All we need to change is the path to the PostgreSQL binaries in the Patroni configuration file. Current I have this:

postgres@patroni3:/home/postgres/ [PG1] grep bin_dir /u01/app/postgres/local/dmk/etc/patroni.yml
  bin_dir: /u01/app/postgres/product/14/db_0/bin

… which points to my PostgreSQL 14.0 installation. After changing that:

postgres@patroni3:/home/postgres/ [PG1] sed -i 's#14/db_0#14/db_1#g' /u01/app/postgres/local/dmk/etc/patroni.yml
postgres@patroni3:/home/postgres/ [PG1] grep bin_dir /u01/app/postgres/local/dmk/etc/patroni.yml
  bin_dir: /u01/app/postgres/product/14/db_1/bin

Once this is done, start the Patroni service and this replica is patched:

postgres@patroni3:/home/postgres/ [PG1] sudo systemctl start patroni
postgres@patroni3:/home/postgres/ [PG1] psql -c "select version()"
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.1 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

Checking the status of the Patroni configuration the replica is back:

postgres@patroni3:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

That was quite easy and you can follow the exact same steps for the second replica. If you prepare that well you’ll have a downtime of a few seconds per replica. The primary, as of now, did not have any downtime. This starts now and you have two choices: If you want that the primary stays on the same host, then you can disable auto failover in Patroni, patch the primary, bring it up again and finally enable auto failover. Another option is to switch the primary to another node, so the current primary becomes a new replica. Once roles are switched you can follow the procedure outlined above.

We’ll do the switchover now:

postgres@patroni2:/home/postgres/ [PG1] patronictl switchover
Master [patroni1]: patroni1
Candidate ['patroni2', 'patroni3'] []: patroni2
When should the switchover take place (e.g. 2021-11-14T19:45 )  [now]: now
Current cluster topology
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Leader  | running |  4 |           |
| patroni2 | 192.168.100.171 | Replica | running |  4 |         0 |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+
Are you sure you want to switchover cluster postgres_ha, demoting current master patroni1? [y/N]: y

2021-11-14 18:45:29.79070 Successfully switched over to "patroni2"
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Replica | stopped |    |   unknown |
| patroni2 | 192.168.100.171 | Leader  | running |  4 |           |
| patroni3 | 192.168.100.172 | Replica | running |  4 |         0 |
+----------+-----------------+---------+---------+----+-----------+

postgres@patroni2:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Replica | running |  5 |         0 |
| patroni2 | 192.168.100.171 | Leader  | running |  5 |           |
| patroni3 | 192.168.100.172 | Replica | running |  5 |         0 |
+----------+-----------------+---------+---------+----+-----------+

Now we can follow the same procedure on the old primary:

postgres@patroni1:/home/postgres/ [PG1] hostname
patroni1
postgres@patroni1:/home/postgres/ [PG1] sudo systemctl stop patroni
postgres@patroni1:/home/postgres/ [PG1] sed -i 's#14/db_0#14/db_1#g' /u01/app/postgres/local/dmk/etc/patroni.yml
postgres@patroni1:/home/postgres/ [PG1] grep bin_dir /u01/app/postgres/local/dmk/etc/patroni.yml
  bin_dir: /u01/app/postgres/product/14/db_1/bin
postgres@patroni1:/home/postgres/ [PG1] sudo systemctl start patroni
postgres@patroni1:/home/postgres/ [PG1] patronictl list
+----------+-----------------+---------+---------+----+-----------+
| Member   | Host            | Role    | State   | TL | Lag in MB |
+ Cluster: postgres_ha (7030448849176438024) ----+----+-----------+
| patroni1 | 192.168.100.170 | Replica | running |  5 |         0 |
| patroni2 | 192.168.100.171 | Leader  | running |  5 |           |
| patroni3 | 192.168.100.172 | Replica | running |  5 |         0 |
+----------+-----------------+---------+---------+----+-----------+
postgres@patroni1:/home/postgres/ [PG1] psql -c "select version()";
                                              version                                              
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.1 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

All done.