Troubleshooting Patroni Cluster: pg_hba Configuration not applied

In the world of database management, high availability and data integrity are of most importance. Patroni, a robust open-source solution, offers a way to manage and maintain PostgreSQL high availability clusters seamlessly. However, even with tools like Patroni, you can face challenges.

The setup: Patroni Cluster Installation

I had the mission of setting up a Patroni cluster for a critical application at a customer’s site. The goal was to ensure continuous database availability, automatic failover, and streamlined management. Patroni’s reputation preceded it, promising a smooth and efficient installation process.

Encountering the edit-config Roadblock

As part of the installation process, I needed to configure PostgreSQL’s `pg_hba.conf` file, which manage client authentication to the databases. To make changes to this file within a Patroni-managed cluster, the command `patronictl edit-config` is typically employed. This command allows for the adjustment of various cluster-related configurations.

However, I was surprised when I attempted to use `patronictl edit-config` to modify the hba configuration. The tool indicated that changes had been made but it wasn’t editing the content of `pg_hba.conf` file. It was a perplexing and frustrating situation, as I was certain that our configuration adjustments weren’t being recognized.

Unraveling the Root Cause

After thorough investigation and consultation with Patroni’s documentation and community, we discovered that the actual issue wasn’t with the `patronictl edit-config` command itself, but rather with our initial configuration approach.

When I had initialized the cluster, I had pre-populated the desired `pg_hba.conf` settings in the `patroni.yml` file under the `postgresql.pg_hba` section. This seemed to be the problem. By including these configurations in advance, I was inadvertently overwriting any subsequent changes made through the Data Center Secret (DCS) mechanism, which `patronictl edit-config` relied on.

The Conflict: `patroni.yml` vs. DCS Changes

In the world of Patroni, the DCS is a critical component responsible for storing and managing cluster state information. This includes configurations like `pg_hba.conf`. When `patronictl edit-config` is employed, it interacts with the DCS to enact changes. However, in our case, our pre-populated `pg_hba.conf` settings in `patroni.yml` were taking precedence over the DCS changes, effectively rendering any subsequent `patronictl edit-config` attempts ineffective.

The Solution: Configuration order matters

Once the root cause was identified, here are the steps to address the issue:

1. Review and Adjust `patroni.yml`: I revisited our `patroni.yml` file and removed the pre-existing `pg_hba.conf` configurations

2. Use `patronictl edit-config`: With the adjustments in place, I employed `patronictl edit-config` to modify the `pg_hba.conf` file as needed. This time, the tool functioned as expected, leveraging the DCS mechanism to enact changes.

3. Cluster Health Verification: After making the necessary `pg_hba.conf` modifications, I tested the cluster’s health to ensure the changes didn’t negatively impact overall functionality.

Another possible root cause

There was another possible root cause: the settings hba_file in the patroni.yml file under the section postgresql.hba_file has been configured.
In case you set this parameter, which should have as a value a path for pg_hba.conf file, the postgres.pg_hba section or the change made with Patroni through DCS will be ignored.
In my case, this parameter was not set.

Conclusion

My encounter with this configuration conflict provided valuable insights into the various way of working with Patroni clusters. Lessons learned from this experience include:

– Configuration Order Matters: The order in which configurations are applied can significantly impact the behavior of cluster management tools like Patroni.

– DCS and `patronictl edit-config`: Understanding the role of the Data Center Secret (DCS) and its interaction with `patronictl edit-config` is crucial for smooth cluster management.

By sharing my experience, I hope to help others in navigating similar hurdles and optimizing their Patroni-managed clusters.