Last week the new Patroni 2.0 release was published which brings many new features. But one makes me absolutely curious. At the moment only as BETA, but I had to test it. Patroni on pure Raft. It’s possible to run Patroni without 3rd party dependencies. So no Etcd, Consul or Zookeeper is needed anymore. Great improvement!
In this blog we will have a look at the setup and try an failover as well.

Starting position

With Patroni on Raft it is possible to run a two node Patroni cluster as well, but I decided to setup a three node cluster.
So what you need to prepare:

  • Three identical VMs with CentOS8 installed
  • All with Postgres 13 and it’s dependencies installed from source. I chose Postgres 13, because the support is newly added as well in Patroni 2.0.
  • I also created the /etc/hostname entries and exchanged the ssh-key between the three servers.
  • Our DMK is installed on these servers as well.
  • Firewall and SELinux are disabled in this example. If you want to run the setup with both enable, you need to configure the firewall and SELinux first.

Setup Patroni

Let’s start with the installation of Patroni. The following steps need to be performed on all three servers. The installation is also not as complicated as before with the new release.

[[email protected] ~]$ yum install python3-psycopg2
[[email protected] ~]$ su - postgres
[email protected]:/home/postgres/ [pg130] pip3 install patroni[raft] --user
Collecting patroni[raft]
  Using cached
Requirement already satisfied: cdiff in /usr/local/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: urllib3[secure]!=1.21,>=1.19.1 in /usr/local/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: PyYAML in /usr/local/lib64/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: six>=1.7 in /usr/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: prettytable>=0.7 in /usr/local/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: python-dateutil in /usr/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: click>=4.1 in /usr/local/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: psutil>=2.0.0 in /usr/local/lib64/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: pysyncobj>=0.3.5; extra == "raft" in /usr/local/lib/python3.6/site-packages (from patroni[raft])
Requirement already satisfied: idna>=2.0.0; extra == "secure" in /usr/lib/python3.6/site-packages (from urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Requirement already satisfied: pyOpenSSL>=0.14; extra == "secure" in /usr/lib/python3.6/site-packages (from urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Requirement already satisfied: certifi; extra == "secure" in /usr/local/lib/python3.6/site-packages (from urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Requirement already satisfied: cryptography>=1.3.4; extra == "secure" in /usr/lib64/python3.6/site-packages (from urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/lib/python3.6/site-packages (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Requirement already satisfied: cffi!=1.11.3,>=1.7 in /usr/lib64/python3.6/site-packages (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Requirement already satisfied: pycparser in /usr/lib/python3.6/site-packages (from cffi!=1.11.3,>=1.7->cryptography>=1.3.4; extra == "secure"->urllib3[secure]!=1.21,>=1.19.1->patroni[raft])
Installing collected packages: patroni
Successfully installed patroni-2.0.0


As the installation was successful, we can go on with the configuration of Patroni. As I am using our DMK, the patroni.yml file is stored in the DMK home. But you can store it somewhere else (of course). You only have to adjust some values like IP addresses and name on every server.
But the most important section in here is the Raft section (line 17-22).
– data_dir: For storing the Raft Log and snapshot. It’s an optional parameter.
– self_addr: It’s the address to listen for Raft connections. This needs to be set. Otherwise the node will not become part of the consensus.
– partner_addrs: Here the list of the other Patroni nodes needs to be added.

[email protected]:/u01/app/postgres/local/dmk/etc/ [pg130] cat patroni.yml
scope: PG1
#namespace: /service/
name: patroni1

#  certfile: /etc/ssl/certs/ssl-cert-snakeoil.pem
#  keyfile: /etc/ssl/private/ssl-cert-snakeoil.key
#  authentication:
#    username: username
#    password: password

# ctl:
#   insecure: false # Allow connections to SSL sites without certs
#   certfile: /etc/ssl/certs/ssl-cert-snakeoil.pem
#   cacert: /etc/ssl/certs/ssl-cacert-snakeoil.pem

  data_dir: /u02/pgdata/raft
  partner_addrs: ['','']

  # and all other cluster members will use it as a `global configuration`
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
      use_pg_rewind: true
      use_slots: true
        wal_level: 'hot_standby'
        hot_standby: "on"
        wal_keep_segments: 8
        max_replication_slots: 10
        wal_log_hints: "on"
        listen_addresses: '*'
        port: 5432
        logging_collector: 'on'
        log_truncate_on_rotation: 'on'
        log_filename: 'postgresql-%a.log'
        log_rotation_age: '1440'
        log_line_prefix: '%m - %l - %p - %h - %[email protected]%d - %x'
        log_directory: 'pg_log'
        log_min_messages: 'WARNING'
        log_autovacuum_min_duration: '60s'
        log_min_error_statement: 'NOTICE'
        log_min_duration_statement: '30s'
        log_checkpoints: 'on'
        log_statement: 'ddl'
        log_lock_waits: 'on'
        log_temp_files: '0'
        log_timezone: 'Europe/Zurich'
        log_connections: 'on'
        log_disconnections: 'on'
        log_duration: 'on'
        client_min_messages: 'WARNING'
        wal_level: 'replica'
        hot_standby_feedback: 'on'
        max_wal_senders: '10'
        shared_buffers: '128MB'
        work_mem: '8MB'
        effective_cache_size: '512MB'
        maintenance_work_mem: '64MB'
        wal_compression: 'off'
        max_wal_senders: '20'
        shared_preload_libraries: 'pg_stat_statements'
        autovacuum_max_workers: '6'
        autovacuum_vacuum_scale_factor: '0.1'
        autovacuum_vacuum_threshold: '50'
        archive_mode: 'on'
        archive_command: '/bin/true'
        wal_log_hints: 'on'
#      recovery_conf:
#        restore_command: cp ../wal_archive/%f %p

  # some desired options for 'initdb'
  initdb:  # Note: It needs to be a list (some options need values, others are switches)
  - encoding: UTF8
  - data-checksums

  pg_hba:  # Add following lines to pg_hba.conf after running 'initdb'
  - host replication replicator md5
  - host all all md5
#  - hostssl all all md5

  # Additional script to be launched after initial cluster creation (will be passed the connection URL as parameter)
# post_init: /usr/local/bin/

  # Some additional users users which needs to be created after initializing new cluster
      password: admin
        - createrole
        - createdb
      password: postgres
        - superuser

  data_dir: /u02/pgdata/13/PG1
  bin_dir: /u01/app/postgres/product/13/db_0/bin
#  config_dir:
  pgpass: /u01/app/postgres/local/dmk/etc/pgpass0
      username: replicator
      password: *********
      username: postgres
      password: *********
    unix_socket_directories: '/tmp'

  mode: automatic # Allowed values: off, automatic, required
  device: /dev/watchdog
  safety_margin: 5

    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false


To start Patroni automatically after reboot. Let’s create a service.

# systemd integration for patroni
# Put this file under /etc/systemd/system/patroni.service
#     then: systemctl daemon-reload
#     then: systemctl list-unit-files | grep patroni
#     then: systemctl enable patroni.service

Description=dbi services patroni service

ExecStartPre=-/usr/bin/sudo /sbin/modprobe softdog
ExecStartPre=-/usr/bin/sudo /bin/chown postgres /dev/watchdog
ExecStart=/home/postgres/.local/bin/patroni /u01/app/postgres/local/dmk/etc/patroni.yml
ExecReload=/bin/kill -s HUP $MAINPID


Once everything is created and before starting Patroni, it is possible to check the Patroni configuration file. And that’s the point, when it gets a bit funny at the moment.

[email protected]:/u02/pgdata/raft/ [PG1] patroni --validate-config /u01/app/postgres/local/dmk/etc/patroni.yml
restapi.listen didn't pass validation: 'Port 8008 is already in use.'
Traceback (most recent call last):
  File "/u01/app/postgres/local/dmk/bin/patroni", line 11, in 
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 170, in main
    return patroni_main()
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 88, in abstract_main
    Config(args.configfile, validator=validator)
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 102, in __init__
    error = validator(self._local_configuration)
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 177, in __call__
    for i in self.validate(data):
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 209, in validate
    for i in self.iter():
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 217, in iter
    for i in self.iter_dict():
  File "/home/postgres/.local/lib/python3.6/site-packages/patroni/", line 244, in iter_dict
    validator = self.validator[key]._schema[d]
KeyError: 'raft'

I tried several version of the Raft configuration. But every time I got an error. I also tried to set the system parameters for Raft and commented the Raft block out in the configuration file. But then I got the following output.

[email protected]:/u02/pgdata/raft/ [PG1] patroni --validate-config /u01/app/postgres/local/dmk/etc/patroni.yml
consul  is not defined.
etcd  is not defined.
etcd3  is not defined.
exhibitor  is not defined.
kubernetes  is not defined.
raft  is not defined.
zookeeper  is not defined.
postgresql.authentication.rewind  is not defined.

So seems like there is something not working correctly when validating the configuration file. In the end I was not sure what to test anymore and so I just tried to start the Patroni service. Full exploratory spirit.

[email protected]:/u02/pgdata/raft/ [PG1] sudo systemctl start patroni
[email protected]:/u02/pgdata/raft/ [PG1] sudo systemctl status patroni
● patroni.service - dbi services patroni service
   Loaded: loaded (/etc/systemd/system/patroni.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-09-10 11:36:40 CEST; 3s ago
  Process: 5232 ExecStartPre=/usr/bin/sudo /bin/chown postgres /dev/watchdog (code=exited, status=0/SUCCESS)
  Process: 5229 ExecStartPre=/usr/bin/sudo /sbin/modprobe softdog (code=exited, status=0/SUCCESS)
 Main PID: 5236 (patroni)
    Tasks: 2 (limit: 11480)
   Memory: 19.7M
   CGroup: /system.slice/patroni.service
           └─5236 /usr/bin/python3.6 /u01/app/postgres/local/dmk/bin/patroni /u01/app/postgres/local/dmk/etc/patroni.yml

Sep 10 11:36:40 partoni1 systemd[1]: Starting dbi services patroni service...
Sep 10 11:36:40 partoni1 sudo[5229]: postgres : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/modprobe softdog
Sep 10 11:36:40 partoni1 sudo[5232]: postgres : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown postgres /dev/watchdog
Sep 10 11:36:40 partoni1 systemd[1]: Started dbi services patroni service.

And….it starts without any errors.
But that did not fully convinced me. Let’s check the Raft setup. Here Patroni delivers a simple command to check the status of the Raft setup.

commit_idx: 62850
[email protected]:/u02/pgdata/raft/ [PG1] syncobj_admin -conn -status
enabled_code_version: 0
last_applied: 62850
leader_commit_idx: 62850
log_len: 31
match_idx_count: 0
next_node_idx_count: 0
partner_node_status_server_192.168.22.202:5010: 2
partner_node_status_server_192.168.22.203:5010: 2
partner_nodes_count: 2
raft_term: 30
readonly_nodes_count: 0
revision: deprecated
self_code_version: 0
state: 0
uptime: 379
version: 0.3.6

And of course we can still check the cluster status itself.

[email protected]:/u02/pgdata/raft/ [PG1] patronictl list
+ Cluster: PG1 (6870147915530980670) -+---------+----+-----------+
| Member   | Host           | Role    | State   | TL | Lag in MB |
| patroni1 | | Replica | running |  6 |         0 |
| patroni2 | | Leader  | running |  6 |           |
| patroni3 | | Replica | running |  6 |         0 |

This look good as well. So even the configuration validation gives us an error, the Cluster is running smoothly.


So let’s see what happens if we restart the Leader node. Within a short time, the Leader changes to another node

[email protected]:/home/postgres/ [PG1] patronictl list
+ Cluster: PG1 (6870147915530980670) -+---------+----+-----------+
| Member   | Host           | Role    | State   | TL | Lag in MB |
| patroni1 | | Leader  | running |  7 |           |
| patroni3 | | Replica | running |  7 |         0 |

As soon as patroni2 is back to the network, it will attach to the cluster again without issues as a Replica

[email protected]:/home/postgres/ [PG1] patronictl list
+ Cluster: PG1 (6870147915530980670) -+---------+----+-----------+
| Member   | Host           | Role    | State   | TL | Lag in MB |
| patroni1 | | Leader  | running |  7 |           |
| patroni2 | | Replica | running |  7 |         0 |
| patroni3 | | Replica | running |  7 |         0 |


The configuration check was published as a first draft in Version 1.6.5. Seems there is some space for improvement here. Still not sure where I do the mistake and if there is really one. I also tested my patroni.yml with is configured for etcd. Maybe it’s just because of the brand new Raft possibility.

For me, the Raft status overview is a bit cryptical. Of course, it shows the most important information, like leader, partner_node_status_server and partner_nodes_count. But compared to etcd, where I just get a simple “cluster is healthy”, it needs a bit time to getting used to. Besides that it needs some time to recognize the unavailability of one node.

Using Patroni with pure Raft works fine and the setup is easier than the etcd setup, where you can get some member mismatches. Especially when you don’t want to install an additional tool on your server, it could get a really good possibility. The documentation of Patroni is still a bit minimalistic regarding the configuration. But with some patience you find whatever you’re searching for.