This blog describes the setup of a Consul Cluster on RHEL 8 and clones, it will be the base for a Patroni HA setup using RPM Packages from postgresql.org.
Many Patroni setups are using ETCD, but ETCD is not available as RPM out of the box for RHEL 8 and clones, and in many cases using tar files or RPMS from unknown sources are not allowed.
I use OS Rocky Linux 8.5 minimal installation patched before writing this blog.
[root@patroni-01 ~]# cat /etc/os-release NAME="Rocky Linux" VERSION="8.5 (Green Obsidian)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="8.5" PLATFORM_ID="platform:el8" PRETTY_NAME="Rocky Linux 8.5 (Green Obsidian)" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:rocky:rocky:8:GA" HOME_URL="https://rockylinux.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" ROCKY_SUPPORT_PRODUCT="Rocky Linux" ROCKY_SUPPORT_PRODUCT_VERSION="8" [root@patroni-01 ~]#
It will be a three node cluster with the following nodes:
[root@patroni-01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 192.168.198.132 patroni-01.patroni.test patroni-01 192.168.198.133 patroni-02.patroni.test patroni-02 192.168.198.134 patroni-03.patroni.test patroni-03 [root@patroni-01 ~]#
The installation RPM for Consul will come from the postgresql.org repository, so we need to disable postgresql from the OS Repository on all three nodes.
$ [root@patroni-01 ~]# dnf -y module disable postgresql $ Last metadata expiration check: 1:21:07 ago on Fri 04 Mar 2022 12:53:22 PM CET. $ Dependencies resolved. $ ==================================================================================================== $ Package Architecture Version Repository Size $ ==================================================================================================== $ Disabling modules: $ postgresql $ $ Transaction Summary $ ==================================================================================================== $ $ Complete! $ [root@patroni-01 ~]#
Next step is adding the postgresql.org repository.
$ [root@patroni-01 ~]# dnf install https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm $ Last metadata expiration check: 1:49:40 ago on Fri 04 Mar 2022 12:53:22 PM CET. $ pgdg-redhat-repo-latest.noarch.rpm 13 kB/s | 12 kB 00:00 $ Dependencies resolved. $ ========================================================================================================================================== $ Package Architecture Version Repository Size $ ========================================================================================================================================== $ Installing: $ pgdg-redhat-repo noarch 42.0-23 @commandline 12 k $ $ Transaction Summary $ ========================================================================================================================================== $ Install 1 Package $ $ Total size: 12 k $ Installed size: 12 k $ Is this ok [y/N]: y $ Downloading Packages: $ Running transaction check $ Transaction check succeeded. $ Running transaction test $ Transaction test succeeded. $ Running transaction $ Preparing : 1/1 $ Installing : pgdg-redhat-repo-42.0-23.noarch 1/1 $ Verifying : pgdg-redhat-repo-42.0-23.noarch 1/1 $ $ Installed: $ pgdg-redhat-repo-42.0-23.noarch $ $ Complete! $ [root@patroni-01 ~]#
As written in the beginning, Consul will be part of a Patroni based HA Cluster, so I install all needed packages.
But first I edit the pgdg repo file to enable PostgreSQL 14 only and disable all other versions.
[root@patroni-01 ~]# cat /etc/yum.repos.d/pgdg-redhat-all.repo ####################################################### # PGDG Red Hat Enterprise Linux / CentOS repositories # ####################################################### # PGDG Red Hat Enterprise Linux / CentOS stable common repository for all PostgreSQL versions [pgdg-common] name=PostgreSQL common RPMs for RHEL/CentOS $releasever - $basearch baseurl=https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-$releasever-$basearch enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG repo_gpgcheck = 1 # Red Hat recently breaks compatibility between 8.n and 8.n+1. PGDG repo is # affected with the LLVM repo. This is a band aid repo for the llvmjit users # whose installations cannot be updated. [pgdg-centos8-sysupdates] name=PostgreSQL Supplementary ucommon RPMs for RHEL/CentOS $releasever - $basearch baseurl=https://download.postgresql.org/pub/repos/yum/common/pgdg-centos8-sysupdates/redhat/rhel-$releasever-$basearch enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG repo_gpgcheck = 1 # PGDG Red Hat Enterprise Linux / CentOS stable repositories: [pgdg14] name=PostgreSQL 14 for RHEL/CentOS $releasever - $basearch baseurl=https://download.postgresql.org/pub/repos/yum/14/redhat/rhel-$releasever-$basearch enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG repo_gpgcheck = 1 [pgdg13] name=PostgreSQL 13 for RHEL/CentOS $releasever - $basearch baseurl=https://download.postgresql.org/pub/repos/yum/13/redhat/rhel-$releasever-$basearch enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG repo_gpgcheck = 1
Set enabled to 0 for all other version than the one you want to install, in my case only 14 is enabled.
As preparation for the following operations we need to open ports with firewalld, port 5432 is default PostgreSQL, the others used by consul.
$ [root@patroni-01 ~]# firewall-cmd --add-port={5432,8300,8301,8302,8400,8500,8600}/tcp --permanent $ success $ [root@patroni-01 ~]# firewall-cmd --add-port={8301,8302,8600}/udp --permanent $ success $ [root@patroni-01 ~]# firewall-cmd --reload $ success $ [root@patroni-01 ~]# $
Now it is time to install Consul, as written in the beginning it will be part of a Patroni Cluster, so I install all needed packages in one step.
$ [root@patroni-01 ~]# dnf install consul postgresql14 postgresql14-server postgresql14-contrib haproxy keepalived patroni $ PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64 83 B/s | 195 B 00:02 $ PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64 1.6 MB/s | 1.7 kB 00:00 $ Importing GPG key 0x442DF0F8: $ Userid : "PostgreSQL RPM Building Project " $ Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8 $ From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG $ Is this ok [y/N]: y $ PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64 186 kB/s | 619 kB 00:03 $ PostgreSQL 14 for RHEL/CentOS 8 - x86_64 129 B/s | 195 B 00:01 $ PostgreSQL 14 for RHEL/CentOS 8 - x86_64 1.6 MB/s | 1.7 kB 00:00 $ Importing GPG key 0x442DF0F8: $ Userid : "PostgreSQL RPM Building Project " $ Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8 $ From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG $ Is this ok [y/N]: y $ PostgreSQL 14 for RHEL/CentOS 8 - x86_64 74 kB/s | 206 kB 00:02 $ Dependencies resolved. $ =========================================================================================================================== $ Package Architecture Version Repository Size $ =========================================================================================================================== $ Installing: $ consul x86_64 1.10.3-1.rhel8 pgdg-common 16 M $ haproxy x86_64 1.8.27-2.el8 appstream 1.4 M $ keepalived x86_64 2.1.5-6.el8 appstream 535 k $ patroni x86_64 2.1.3-1.rhel8 pgdg-common 863 k $ postgresql14 x86_64 14.2-1PGDG.rhel8 pgdg14 1.5 M $ postgresql14-contrib x86_64 14.2-1PGDG.rhel8 pgdg14 723 k $ postgresql14-server x86_64 14.2-1PGDG.rhel8 pgdg14 5.7 M $ Installing dependencies: $ libicu x86_64 60.3-2.el8_1 baseos 8.8 M $ lm_sensors-libs x86_64 3.4.0-23.20180522git70f7e08.el8 baseos 58 k $ lz4 x86_64 1.8.3-3.el8_4 baseos 102 k $ mariadb-connector-c x86_64 3.1.11-2.el8_3 appstream 199 k $ mariadb-connector-c-config noarch 3.1.11-2.el8_3 appstream 14 k $ net-snmp-agent-libs x86_64 1:5.8-22.el8 appstream 747 k $ net-snmp-libs x86_64 1:5.8-22.el8 baseos 826 k $ perl-Carp noarch 1.42-396.el8 baseos 29 k $ perl-Data-Dumper x86_64 2.167-399.el8 baseos 57 k $ perl-Digest noarch 1.17-395.el8 appstream 26 k $ perl-Digest-MD5 x86_64 2.55-396.el8 appstream 36 k $ perl-Encode x86_64 4:2.97-3.el8 baseos 1.5 M $ perl-Errno x86_64 1.28-420.el8 baseos 75 k $ perl-Exporter noarch 5.72-396.el8 baseos 33 k $ perl-File-Path noarch 2.15-2.el8 baseos 37 k $ perl-File-Temp noarch 0.230.600-1.el8 baseos 62 k $ perl-Getopt-Long noarch 1:2.50-4.el8 baseos 62 k $ perl-HTTP-Tiny noarch 0.074-1.el8 baseos 57 k $ perl-IO x86_64 1.38-420.el8 baseos 141 k $ perl-MIME-Base64 x86_64 3.15-396.el8 baseos 30 k $ perl-Net-SSLeay x86_64 1.88-1.module+el8.4.0+512+d4f0fc54 appstream 378 k $ perl-PathTools x86_64 3.74-1.el8 baseos 89 k $ perl-Pod-Escapes noarch 1:1.07-395.el8 baseos 19 k $ perl-Pod-Perldoc noarch 3.28-396.el8 baseos 85 k $ perl-Pod-Simple noarch 1:3.35-395.el8 baseos 212 k $ perl-Pod-Usage noarch 4:1.69-395.el8 baseos 33 k $ perl-Scalar-List-Utils x86_64 3:1.49-2.el8 baseos 67 k $ perl-Socket x86_64 4:2.027-3.el8 baseos 58 k $ perl-Storable x86_64 1:3.11-3.el8 baseos 97 k $ perl-Term-ANSIColor noarch 4.06-396.el8 baseos 45 k $ perl-Term-Cap noarch 1.17-395.el8 baseos 22 k $ perl-Text-ParseWords noarch 3.30-395.el8 baseos 17 k $ perl-Text-Tabs+Wrap noarch 2013.0523-395.el8 baseos 23 k $ perl-Time-Local noarch 1:1.280-1.el8 baseos 32 k $ perl-URI noarch 1.73-3.el8 appstream 115 k $ perl-Unicode-Normalize x86_64 1.25-396.el8 baseos 81 k $ perl-constant noarch 1.33-396.el8 baseos 24 k $ perl-interpreter x86_64 4:5.26.3-420.el8 baseos 6.3 M $ perl-libnet noarch 3.11-3.el8 appstream 120 k $ perl-libs x86_64 4:5.26.3-420.el8 baseos 1.6 M $ perl-macros x86_64 4:5.26.3-420.el8 baseos 71 k $ perl-parent noarch 1:0.237-1.el8 baseos 19 k $ perl-podlators noarch 4.11-1.el8 baseos 117 k $ perl-threads x86_64 1:2.21-2.el8 baseos 60 k $ perl-threads-shared x86_64 1.58-2.el8 baseos 47 k $ postgresql14-libs x86_64 14.2-1PGDG.rhel8 pgdg14 275 k $ python3-cdiff noarch 1.0-1.rhel8 pgdg-common 30 k $ python3-click noarch 6.7-8.el8 appstream 130 k $ python3-pip noarch 9.0.3-20.el8.rocky.0 appstream 19 k $ python3-prettytable noarch 0.7.2-14.el8 appstream 43 k $ python3-psutil x86_64 5.4.3-11.el8 appstream 372 k $ python3-psycopg2 x86_64 2.8.6-1.rhel8 pgdg-common 178 k $ python3-pyyaml x86_64 3.12-12.el8 baseos 192 k $ python3-setuptools noarch 39.2.0-6.el8 baseos 162 k $ python3-ydiff noarch 1.2-10.rhel8 pgdg-common 30 k $ python36 x86_64 3.6.8-38.module+el8.5.0+671+195e4563 appstream 18 k $ Installing weak dependencies: $ perl-IO-Socket-IP noarch 0.39-5.el8 appstream 46 k $ perl-IO-Socket-SSL noarch 2.066-4.module+el8.4.0+512+d4f0fc54 appstream 297 k $ perl-Mozilla-CA noarch 20160104-7.module+el8.4.0+529+e3b3e624 appstream 14 k $ Enabling module streams: $ perl 5.26 $ perl-IO-Socket-SSL 2.066 $ perl-libwww-perl 6.34 $ python36 3.6 $ $ Transaction Summary $ =========================================================================================================================== $ Install 66 Packages $ $ Total download size: 51 M $ Installed size: 203 M $ Is this ok [y/N]:
With installation out of the postgresql.org repository also user postgres and group postgres are created.
I want Consul also to run as postgres user, for this we need to adapt User and Group within the service file.
The service file is located at /usr/lib/systemd/system/consul.service.
[root@patroni-01 ~]# cat /usr/lib/systemd/system/consul.service [Unit] Description=Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable. Documentation=http://www.consul.io After=network-online.target Wants=network-online.target [Service] User=postgres Group=postgres EnvironmentFile=-/etc/sysconfig/consul ExecStart=/usr/bin/consul $CMD_OPTS ExecReload=/bin/kill -HUP $MAINPID KillSignal=SIGINT Restart=on-failure [Install] WantedBy=multi-user.target [root@patroni-01 ~]#
Next step is adapting the Consul environment file, I want the Consul data directory within /pgdata.
The environment file is located at /etc/sysconfig/consul.
[root@patroni-01 ~]# cat /etc/sysconfig/consul CMD_OPTS="agent -config-dir=/etc/consul.d -data-dir=/pgdata/consul" #GOMAXPROCS=4 [root@patroni-01 ~]#
Creating Consul data directory.
$ [root@patroni-01 ~]# mkdir /pgdata/consul $ [root@patroni-01 ~]# chown -R postgres:postgres /pgdata/
And the Consul key.
$ [root@patroni-01 ~]# consul keygen $ 5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE= $ [root@patroni-01 ~]#
Now it is time to adapt the Consul configuration file on each node.
Node patroni-01:
[root@patroni-01 ~]# cat /etc/consul.d/consul.json-dist.hcl { "server": true, "data_dir": "/pgdata/consul", "log_level": "INFO" "disable_update_check": true, "disable_anonymous_signature": true, "advertise_addr": "192.168.198.132", "bind_addr": "192.168.198.132", "bootstrap_expect": 3, "client_addr": "0.0.0.0", "domain": "patroni.test", "enable_script_checks": true, "dns_config": { "enable_truncate": true, "only_passing": true }, "enable_syslog": true, "encrypt": "5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=", "leave_on_terminate": true, "log_level": "INFO", "rejoin_after_leave": true, "retry_join": [ "patroni-01", "patroni-02", "patroni-03" ], "server": true, "start_join": [ "patroni-01", "patroni-02", "patroni-03" ], "ui_config.enabled": true } [root@patroni-01 ~]#
Node patroni-02:
[root@patroni-02 ~]# cat /etc/consul.d/consul.json-dist.hcl { "server": true, "data_dir": "/pgdata/consul", "log_level": "INFO" "disable_update_check": true, "disable_anonymous_signature": true, "advertise_addr": "192.168.198.133", "bind_addr": "192.168.198.133", "bootstrap_expect": 3, "client_addr": "0.0.0.0", "domain": "patroni.test", "enable_script_checks": true, "dns_config": { "enable_truncate": true, "only_passing": true }, "enable_syslog": true, "encrypt": "5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=", "leave_on_terminate": true, "log_level": "INFO", "rejoin_after_leave": true, "retry_join": [ "patroni-01", "patroni-02", "patroni-03" ], "server": true, "start_join": [ "patroni-01", "patroni-02", "patroni-03" ], "ui_config.enabled": true } [root@patroni-02 ~]#
Node patroni-03:
[root@patroni-03 ~]# cat /etc/consul.d/consul.json-dist.hcl { "server": true, "data_dir": "/pgdata/consul", "log_level": "INFO" "disable_update_check": true, "disable_anonymous_signature": true, "advertise_addr": "192.168.198.134", "bind_addr": "192.168.198.134", "bootstrap_expect": 3, "client_addr": "0.0.0.0", "domain": "patroni.test", "enable_script_checks": true, "dns_config": { "enable_truncate": true, "only_passing": true }, "enable_syslog": true, "encrypt": "5mSUIrSSXp+usVR1qqM68CD2lnFLaTcg4G48l9zJhqE=", "leave_on_terminate": true, "log_level": "INFO", "rejoin_after_leave": true, "retry_join": [ "patroni-01", "patroni-02", "patroni-03" ], "server": true, "start_join": [ "patroni-01", "patroni-02", "patroni-03" ], "ui_config.enabled": true } [root@patroni-03 ~]#
And make the files accessable for the postgres user.
$ [root@patroni-01 ~]# chown -R postgres:postgres /etc/consul.d/
Now it is time to start Consul on each node.
$ [root@patroni-01 ~]# systemctl start consul
Checking the status.
[root@patroni-01 ~]# consul members $ Node Address Status Type Build Protocol DC Segment $ patroni-01.patroni.test 192.168.198.132:8301 alive server 1.10.3 2 dc1 $ patroni-02.patroni.test 192.168.198.133:8301 alive server 1.10.3 2 dc1 $ patroni-03.patroni.test 192.168.198.134:8301 alive server 1.10.3 2 dc1 $ [root@patroni-01 ~]#
Sometimes it happens that Consul autojoin has issues, in this case manual join helps.
$ [root@patroni-01 ~]# consul join 192.168.198.132 192.168.198.133 192.168.198.134 $ Successfully joined cluster by contacting 3 nodes. $ [root@patroni-01 ~]# consul members $ Node Address Status Type Build Protocol DC Segment $ patroni-01.patroni.test 192.168.198.132:8301 alive server 1.10.3 2 dc1 $ patroni-02.patroni.test 192.168.198.133:8301 alive server 1.10.3 2 dc1 $ patroni-03.patroni.test 192.168.198.134:8301 alive server 1.10.3 2 dc1 $ [root@patroni-01 ~]#
That was the first part of a Patroni HA Setup using Consul instead of ETCD.