This blog describes the setup of a Consul Cluster on RHEL 8 and clones, it will be the base for a Patroni HA setup using RPM Packages from postgresql.org.
Many Patroni setups are using ETCD, but ETCD is not available as RPM out of the box for RHEL 8 and clones, and in many cases using tar files or RPMS from unknown sources are not allowed.

I use OS Rocky Linux 8.5 minimal installation patched before writing this blog.

[[email protected] ~]# cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.5 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.5 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
[[email protected] ~]#

It will be a three node cluster with the following nodes:

[[email protected] ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
192.168.198.132 patroni-01.patroni.test patroni-01
192.168.198.133 patroni-02.patroni.test patroni-02
192.168.198.134 patroni-03.patroni.test patroni-03
[[email protected] ~]#

The installation RPM for Consul will come from the postgresql.org repository, so we need to disable postgresql from the OS Repository on all three nodes.

$ [[email protected] ~]# dnf -y module disable postgresql
$ Last metadata expiration check: 1:21:07 ago on Fri 04 Mar 2022 12:53:22 PM CET.
$ Dependencies resolved.
$ ====================================================================================================
$  Package                Architecture          Version                  Repository              Size
$ ====================================================================================================
$ Disabling modules:
$  postgresql
$ 
$ Transaction Summary
$ ====================================================================================================
$ 
$ Complete!
$ [[email protected] ~]#

Next step is adding the postgresql.org repository.

$ [[email protected] ~]# dnf install https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
$ Last metadata expiration check: 1:49:40 ago on Fri 04 Mar 2022 12:53:22 PM CET.
$ pgdg-redhat-repo-latest.noarch.rpm                                                                         13 kB/s |  12 kB     00:00
$ Dependencies resolved.
$ ==========================================================================================================================================
$  Package                               Architecture                Version                        Repository                         Size
$ ==========================================================================================================================================
$ Installing:
$  pgdg-redhat-repo                      noarch                      42.0-23                        @commandline                       12 k
$ 
$ Transaction Summary
$ ==========================================================================================================================================
$ Install  1 Package
$ 
$ Total size: 12 k
$ Installed size: 12 k
$ Is this ok [y/N]: y
$ Downloading Packages:
$ Running transaction check
$ Transaction check succeeded.
$ Running transaction test
$ Transaction test succeeded.
$ Running transaction
$   Preparing        :                                                                                                                  1/1
$   Installing       : pgdg-redhat-repo-42.0-23.noarch                                                                                  1/1
$   Verifying        : pgdg-redhat-repo-42.0-23.noarch                                                                                  1/1
$ 
$ Installed:
$   pgdg-redhat-repo-42.0-23.noarch
$ 
$ Complete!
$ [[email protected] ~]#

As written in the beginning, Consul will be part of a Patroni based HA Cluster, so I install all needed packages.
But first I edit the pgdg repo file to enable PostgreSQL 14 only and disable all other versions.

[[email protected] ~]# cat /etc/yum.repos.d/pgdg-redhat-all.repo
#######################################################
# PGDG Red Hat Enterprise Linux / CentOS repositories #
#######################################################

# PGDG Red Hat Enterprise Linux / CentOS stable common repository for all PostgreSQL versions

[pgdg-common]
name=PostgreSQL common RPMs for RHEL/CentOS $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-$releasever-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
repo_gpgcheck = 1

# Red Hat recently breaks compatibility between 8.n and 8.n+1. PGDG repo is
# affected with the LLVM repo. This is a band aid repo for the llvmjit users
# whose installations cannot be updated.

[pgdg-centos8-sysupdates]
name=PostgreSQL Supplementary ucommon RPMs for RHEL/CentOS $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/common/pgdg-centos8-sysupdates/redhat/rhel-$releasever-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
repo_gpgcheck = 1

# PGDG Red Hat Enterprise Linux / CentOS stable repositories:

[pgdg14]
name=PostgreSQL 14 for RHEL/CentOS $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/14/redhat/rhel-$releasever-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
repo_gpgcheck = 1

[pgdg13]
name=PostgreSQL 13 for RHEL/CentOS $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/13/redhat/rhel-$releasever-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
repo_gpgcheck = 1

Set enabled to 0 for all other version than the one you want to install, in my case only 14 is enabled.

As preparation for the following operations we need to open ports with firewalld, port 5432 is default PostgreSQL, the others used by consul.

$ [[email protected] ~]# firewall-cmd --add-port={5432,8300,8301,8302,8400,8500,8600}/tcp --permanent
$ success
$ [[email protected] ~]# firewall-cmd --add-port={8301,8302,8600}/udp --permanent
$ success
$ [[email protected] ~]# firewall-cmd --reload
$ success
$ [[email protected] ~]#
$ 

Now it is time to install Consul, as written in the beginning it will be part of a Patroni Cluster, so I install all needed packages in one step.

$ [[email protected] ~]# dnf install consul postgresql14 postgresql14-server postgresql14-contrib haproxy keepalived patroni
$ PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                           83  B/s | 195  B     00:02
$ PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                          1.6 MB/s | 1.7 kB     00:00
$ Importing GPG key 0x442DF0F8:
$  Userid     : "PostgreSQL RPM Building Project "
$  Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
$  From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
$ Is this ok [y/N]: y
$ PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64                                          186 kB/s | 619 kB     00:03
$ PostgreSQL 14 for RHEL/CentOS 8 - x86_64                                                   129  B/s | 195  B     00:01
$ PostgreSQL 14 for RHEL/CentOS 8 - x86_64                                                   1.6 MB/s | 1.7 kB     00:00
$ Importing GPG key 0x442DF0F8:
$  Userid     : "PostgreSQL RPM Building Project "
$  Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8
$  From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG
$ Is this ok [y/N]: y
$ PostgreSQL 14 for RHEL/CentOS 8 - x86_64                                                    74 kB/s | 206 kB     00:02
$ Dependencies resolved.
$ ===========================================================================================================================
$  Package                           Architecture  Version                                          Repository          Size
$ ===========================================================================================================================
$ Installing:
$  consul                            x86_64        1.10.3-1.rhel8                                   pgdg-common         16 M
$  haproxy                           x86_64        1.8.27-2.el8                                     appstream          1.4 M
$  keepalived                        x86_64        2.1.5-6.el8                                      appstream          535 k
$  patroni                           x86_64        2.1.3-1.rhel8                                    pgdg-common        863 k
$  postgresql14                      x86_64        14.2-1PGDG.rhel8                                 pgdg14             1.5 M
$  postgresql14-contrib              x86_64        14.2-1PGDG.rhel8                                 pgdg14             723 k
$  postgresql14-server               x86_64        14.2-1PGDG.rhel8                                 pgdg14             5.7 M
$ Installing dependencies:
$  libicu                            x86_64        60.3-2.el8_1                                     baseos             8.8 M
$  lm_sensors-libs                   x86_64        3.4.0-23.20180522git70f7e08.el8                  baseos              58 k
$  lz4                               x86_64        1.8.3-3.el8_4                                    baseos             102 k
$  mariadb-connector-c               x86_64        3.1.11-2.el8_3                                   appstream          199 k
$  mariadb-connector-c-config        noarch        3.1.11-2.el8_3                                   appstream           14 k
$  net-snmp-agent-libs               x86_64        1:5.8-22.el8                                     appstream          747 k
$  net-snmp-libs                     x86_64        1:5.8-22.el8                                     baseos             826 k
$  perl-Carp                         noarch        1.42-396.el8                                     baseos              29 k
$  perl-Data-Dumper                  x86_64        2.167-399.el8                                    baseos              57 k
$  perl-Digest                       noarch        1.17-395.el8                                     appstream           26 k
$  perl-Digest-MD5                   x86_64        2.55-396.el8                                     appstream           36 k
$  perl-Encode                       x86_64        4:2.97-3.el8                                     baseos             1.5 M
$  perl-Errno                        x86_64        1.28-420.el8                                     baseos              75 k
$  perl-Exporter                     noarch        5.72-396.el8                                     baseos              33 k
$  perl-File-Path                    noarch        2.15-2.el8                                       baseos              37 k
$  perl-File-Temp                    noarch        0.230.600-1.el8                                  baseos              62 k
$  perl-Getopt-Long                  noarch        1:2.50-4.el8                                     baseos              62 k
$  perl-HTTP-Tiny                    noarch        0.074-1.el8                                      baseos              57 k
$  perl-IO                           x86_64        1.38-420.el8                                     baseos             141 k
$  perl-MIME-Base64                  x86_64        3.15-396.el8                                     baseos              30 k
$  perl-Net-SSLeay                   x86_64        1.88-1.module+el8.4.0+512+d4f0fc54               appstream          378 k
$  perl-PathTools                    x86_64        3.74-1.el8                                       baseos              89 k
$  perl-Pod-Escapes                  noarch        1:1.07-395.el8                                   baseos              19 k
$  perl-Pod-Perldoc                  noarch        3.28-396.el8                                     baseos              85 k
$  perl-Pod-Simple                   noarch        1:3.35-395.el8                                   baseos             212 k
$  perl-Pod-Usage                    noarch        4:1.69-395.el8                                   baseos              33 k
$  perl-Scalar-List-Utils            x86_64        3:1.49-2.el8                                     baseos              67 k
$  perl-Socket                       x86_64        4:2.027-3.el8                                    baseos              58 k
$  perl-Storable                     x86_64        1:3.11-3.el8                                     baseos              97 k
$  perl-Term-ANSIColor               noarch        4.06-396.el8                                     baseos              45 k
$  perl-Term-Cap                     noarch        1.17-395.el8                                     baseos              22 k
$  perl-Text-ParseWords              noarch        3.30-395.el8                                     baseos              17 k
$  perl-Text-Tabs+Wrap               noarch        2013.0523-395.el8                                baseos              23 k
$  perl-Time-Local                   noarch        1:1.280-1.el8                                    baseos              32 k
$  perl-URI                          noarch        1.73-3.el8                                       appstream          115 k
$  perl-Unicode-Normalize            x86_64        1.25-396.el8                                     baseos              81 k
$  perl-constant                     noarch        1.33-396.el8                                     baseos              24 k
$  perl-interpreter                  x86_64        4:5.26.3-420.el8                                 baseos             6.3 M
$  perl-libnet                       noarch        3.11-3.el8                                       appstream          120 k
$  perl-libs                         x86_64        4:5.26.3-420.el8                                 baseos             1.6 M
$  perl-macros                       x86_64        4:5.26.3-420.el8                                 baseos              71 k
$  perl-parent                       noarch        1:0.237-1.el8                                    baseos              19 k
$  perl-podlators                    noarch        4.11-1.el8                                       baseos             117 k
$  perl-threads                      x86_64        1:2.21-2.el8                                     baseos              60 k
$  perl-threads-shared               x86_64        1.58-2.el8                                       baseos              47 k
$  postgresql14-libs                 x86_64        14.2-1PGDG.rhel8                                 pgdg14             275 k
$  python3-cdiff                     noarch        1.0-1.rhel8                                      pgdg-common         30 k
$  python3-click                     noarch        6.7-8.el8                                        appstream          130 k
$  python3-pip                       noarch        9.0.3-20.el8.rocky.0                             appstream           19 k
$  python3-prettytable               noarch        0.7.2-14.el8                                     appstream           43 k
$  python3-psutil                    x86_64        5.4.3-11.el8                                     appstream          372 k
$  python3-psycopg2                  x86_64        2.8.6-1.rhel8                                    pgdg-common        178 k
$  python3-pyyaml                    x86_64        3.12-12.el8                                      baseos             192 k
$  python3-setuptools                noarch        39.2.0-6.el8                                     baseos             162 k
$  python3-ydiff                     noarch        1.2-10.rhel8                                     pgdg-common         30 k
$  python36                          x86_64        3.6.8-38.module+el8.5.0+671+195e4563             appstream           18 k
$ Installing weak dependencies:
$  perl-IO-Socket-IP                 noarch        0.39-5.el8                                       appstream           46 k
$  perl-IO-Socket-SSL                noarch        2.066-4.module+el8.4.0+512+d4f0fc54              appstream          297 k
$  perl-Mozilla-CA                   noarch        20160104-7.module+el8.4.0+529+e3b3e624           appstream           14 k
$ Enabling module streams:
$  perl                                            5.26
$  perl-IO-Socket-SSL                              2.066
$  perl-libwww-perl                                6.34
$  python36                                        3.6
$ 
$ Transaction Summary
$ ===========================================================================================================================
$ Install  66 Packages
$ 
$ Total download size: 51 M
$ Installed size: 203 M
$ Is this ok [y/N]:

With installation out of the postgresql.org repository also user postgres and group postgres are created.

I want Consul also to run as postgres user, for this we need to adapt User and Group within the service file.
The service file is located at /usr/lib/systemd/system/consul.service.

[[email protected] ~]# cat /usr/lib/systemd/system/consul.service
[Unit]
Description=Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.
Documentation=http://www.consul.io
After=network-online.target
Wants=network-online.target

[Service]
User=postgres
Group=postgres
EnvironmentFile=-/etc/sysconfig/consul
ExecStart=/usr/bin/consul $CMD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGINT
Restart=on-failure

[Install]
WantedBy=multi-user.target
[[email protected] ~]#

Next step is adapting the Consul environment file, I want the Consul data directory within /pgdata.
The environment file is located at /etc/sysconfig/consul.

[[email protected] ~]# cat /etc/sysconfig/consul
CMD_OPTS="agent -config-dir=/etc/consul.d -data-dir=/pgdata/consul"
#GOMAXPROCS=4
[[email protected] ~]#

Creating Consul data directory.

$ [[email protected] ~]# mkdir /pgdata/consul
$ [[email protected] ~]# chown -R postgres:postgres /pgdata/

And the Consul key.

$ [[email protected] ~]# consul keygen
$ 5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=
$ [[email protected] ~]#

Now it is time to adapt the Consul configuration file on each node.
Node patroni-01:

[[email protected] ~]# cat /etc/consul.d/consul.json-dist.hcl
{
    "server": true,
    "data_dir": "/pgdata/consul",
    "log_level": "INFO"
    "disable_update_check": true,
    "disable_anonymous_signature": true,
    "advertise_addr": "192.168.198.132",
    "bind_addr": "192.168.198.132",
    "bootstrap_expect": 3,
    "client_addr": "0.0.0.0",
    "domain": "patroni.test",
    "enable_script_checks": true,
    "dns_config": {
        "enable_truncate": true,
        "only_passing": true
    },
    "enable_syslog": true,
    "encrypt": "5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=",
    "leave_on_terminate": true,
    "log_level": "INFO",
    "rejoin_after_leave": true,
    "retry_join": [
        "patroni-01",
        "patroni-02",
        "patroni-03"
    ],
    "server": true,
    "start_join": [
        "patroni-01",
        "patroni-02",
        "patroni-03"
    ],
    "ui_config.enabled": true
}
[[email protected] ~]#

Node patroni-02:

[[email protected] ~]# cat /etc/consul.d/consul.json-dist.hcl
{
    "server": true,
    "data_dir": "/pgdata/consul",
    "log_level": "INFO"
    "disable_update_check": true,
    "disable_anonymous_signature": true,
    "advertise_addr": "192.168.198.133",
    "bind_addr": "192.168.198.133",
    "bootstrap_expect": 3,
    "client_addr": "0.0.0.0",
    "domain": "patroni.test",
    "enable_script_checks": true,
    "dns_config": {
        "enable_truncate": true,
        "only_passing": true
    },
    "enable_syslog": true,
    "encrypt": "5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=",
    "leave_on_terminate": true,
    "log_level": "INFO",
    "rejoin_after_leave": true,
    "retry_join": [
        "patroni-01",
        "patroni-02",
        "patroni-03"
    ],
    "server": true,
    "start_join": [
        "patroni-01",
        "patroni-02",
        "patroni-03"
    ],
    "ui_config.enabled": true
}
[[email protected] ~]#

Node patroni-03:

[[email protected] ~]# cat /etc/consul.d/consul.json-dist.hcl
{
    "server": true,
    "data_dir": "/pgdata/consul",
    "log_level": "INFO"
    "disable_update_check": true,
    "disable_anonymous_signature": true,
    "advertise_addr": "192.168.198.134",
    "bind_addr": "192.168.198.134",
    "bootstrap_expect": 3,
    "client_addr": "0.0.0.0",
    "domain": "patroni.test",
    "enable_script_checks": true,
    "dns_config": {
        "enable_truncate": true,
        "only_passing": true
    },
    "enable_syslog": true,
    "encrypt": "5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=",
    "leave_on_terminate": true,
    "log_level": "INFO",
    "rejoin_after_leave": true,
    "retry_join": [
        "patroni-01",
        "patroni-02",
        "patroni-03"
    ],
    "server": true,
    "start_join": [
        "patroni-01",
        "patroni-02",
        "patroni-03"
    ],
    "ui_config.enabled": true
}
[[email protected] ~]#

And make the files accessable for the postgres user.

$ [[email protected] ~]# chown -R postgres:postgres /etc/consul.d/

Now it is time to start Consul on each node.

$ [[email protected] ~]# systemctl start consul

Checking the status.

[[email protected] ~]# consul members
$ Node                     Address               Status  Type    Build   Protocol  DC   Segment
$ patroni-01.patroni.test  192.168.198.132:8301  alive   server  1.10.3  2         dc1  
$ patroni-02.patroni.test  192.168.198.133:8301  alive   server  1.10.3  2         dc1  
$ patroni-03.patroni.test  192.168.198.134:8301  alive   server  1.10.3  2         dc1  
$ [[email protected] ~]#

Sometimes it happens that Consul autojoin has issues, in this case manual join helps.

$ [[email protected] ~]# consul join 192.168.198.132 192.168.198.133 192.168.198.134
$ Successfully joined cluster by contacting 3 nodes.
$ [[email protected] ~]# consul members
$ Node                     Address               Status  Type    Build   Protocol  DC   Segment
$ patroni-01.patroni.test  192.168.198.132:8301  alive   server  1.10.3  2         dc1  
$ patroni-02.patroni.test  192.168.198.133:8301  alive   server  1.10.3  2         dc1  
$ patroni-03.patroni.test  192.168.198.134:8301  alive   server  1.10.3  2         dc1  
$ [[email protected] ~]#

That was the first part of a Patroni HA Setup using Consul instead of ETCD.


Thumbnail [60x60]
by
Karsten Lenz