Upgrading from RHEL 9.6 to 10.1 is not just a routine update, it’s a major platform shift. When your server runs PostgreSQL compiled from source and a Patroni-managed cluster, the complexity increases significantly. System libraries change, Python environments break, ICU versions evolve, and your database binaries may no longer start after reboot.

In this guide, I walk through a real-world in-place upgrade using Leapp, covering preparation, resolving high-severity warnings, executing the upgrade, recompiling PostgreSQL, fixing collation mismatches, and restoring Patroni.

I. Preparation

Before the upgrade, you must ensure the current OS is healthy and fully patched.

1. Pause High Availability

Prevent Patroni from triggering a failover during the reboot cycles. If you are using a single PostgreSQL cluster, stop it by stopping the service or by using pg_ctl stop.

patronictl -c /etc/patroni/patroni.yml pause
systemctl stop patroni

2. Fix Subscription & Perform Full Update

I’m using an old VM for this blog, and If just like me, you see 403 Forbidden errors on repositories like codeready-builder, refresh your registration:

[root@patroni2 ~]# sudo dnf update -y
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs)                                                                                                     761  B/s | 480  B     00:00
Errors during downloading metadata for repository 'codeready-builder-for-rhel-9-x86_64-rpms':
  - Status code: 403 for https://cdn.redhat.com/content/dist/rhel9/9/x86_64/codeready-builder/os/repodata/repomd.xml (IP: 23.206.57.92)
Error: Failed to download metadata for repo 'codeready-builder-for-rhel-9-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[root@patroni2 ~]# sudo subscription-manager clean
[root@patroni2 ~]# sudo subscription-manager register --force
[root@patroni2 ~]# sudo subscription-manager attach --auto
[root@patroni2 ~]# sudo subscription-manager refresh

[root@patroni2 ~]# sudo dnf update -y
...
Complete!

[root@patroni2 ~]# reboot

II. The Leapp Upgrade to 10.1

1. Install & Analyze

In this first phase, we are preparing the system for a major in-place upgrade using Leapp, the official upgrade framework for Red Hat–based distributions. When we install the package:

[root@patroni2 ~]# dnf install leapp-upgrade -y
...
Installed:
  leapp-0.20.0-1.el9.noarch                leapp-deps-0.20.0-1.el9.noarch          leapp-upgrade-el9toel10-0.23.0-1.el9.noarch       leapp-upgrade-el9toel10-deps-0.23.0-1.el9.noarch
  libdb-utils-5.3.28-57.el9_6.x86_64       python3-leapp-0.20.0-1.el9.noarch       systemd-container-252-55.el9_7.7.x86_64

Complete!

When running leapp preupgrade –target 10.1, we are not performing the upgrade. Instead, Leapp performs a full system audit to determine if the server is ready for RHEL 10.1. It checks:

  • Installed packages and their compatibility
  • Deprecated or removed libraries
  • Kernel drivers that will not exist in RHEL 10
  • Bootloader configuration (GRUB2)
  • GPG key validity
  • Custom system-level modifications (like dynamic linker changes)
  • …

Think of this step as a dry-run with intelligence.

[root@patroni2 ~]# sudo leapp preupgrade --target 10.1

...

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. GRUB2 core will be automatically updated during the upgrade
    2. Detected customized configuration for dynamic linker.
    3. Leapp detected loaded kernel drivers which are no longer maintained in RHEL 10.
    4. Failed to read GPG keys from provided key files
    5. Berkeley DB (libdb) has been detected on your system

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       4
    MEDIUM severity reports:     1
    LOW severity reports:        1
    INFO severity reports:       3

Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
    A report has been generated at /var/log/leapp/leapp-report.txt
    A report has been generated at /var/log/leapp/leapp-report.json

After running the pre-upgrade analysis, the next step is to carefully review:

/var/log/leapp/leapp-report.txt

What we are looking for first is simple:

  • Errors: 0
  • Inhibitors: 0

If an Inhibitor is present, the upgrade will be blocked entirely.
In my case, there were no blockers, but I did have several high severity warnings.

High severity does not mean the upgrade will fail.
It means: This could break something, review it carefully.

Let’s look at one concrete example from my system.

2. High Severity Example – Dynamic Linker Customization

Leapp detected that my system had a custom dynamic linker configuration:

Risk Factor: high

Title: Detected customized configuration for dynamic linker.

Summary: Custom configurations to the dynamic linker could potentially impact the upgrade in a negative way. The custom configuration includes modifications to /etc/ld.so.conf, custom or modified drop in config files in the /etc/ld.so.conf.d directory and additional entries in the LD_LIBRARY_PATH or LD_PRELOAD variables. These modifications configure the dynamic linker to use different libraries that might not be provided by Red Hat products or might not be present during the whole upgrade process. The following custom configurations were detected by leapp:

- The following drop in config files were marked as custom:

    - /etc/ld.so.conf.d/postgres.conf

Remediation: [hint] Remove or revert the custom dynamic linker configurations and apply the changes using the ldconfig command. In case of possible active software collections we suggest disabling them persistently.

Key: cc9bd972af70b7a27f66a37b11a00dcfcb73b1bc

----------------------------------------

What does this actually mean?

The dynamic linker (ld.so) is responsible for loading shared libraries at runtime.

By modifying:

  • /etc/ld.so.conf
  • files in /etc/ld.so.conf.d/
  • LD_LIBRARY_PATH
  • LD_PRELOAD

we are telling the system to load non-standard or custom libraries. In PostgreSQL environments (especially with custom builds or extensions), this is common practice. However, during a major OS upgrade, these custom paths might:

  • Point to libraries that do not exist in RHEL 10
  • Override new system libraries
  • Break dependency resolution mid-upgrade

Leapp flags this because it cannot guarantee consistency during the transition phase. In my case, it shouldn’t be an issue, because inside postgres.conf, I only have a path aiming to the lib directories of my PostgreSQL installation, which will not change, but we will still see how to prevent an error.

Understanding the Remediation

The report clearly suggests:

Remove or revert the custom dynamic linker configurations and apply the changes using the ldconfig command.

In my case, the configuration was related to PostgreSQL, so temporarily removing it is safe for the upgrade preparation phase. Instead of deleting it permanently, I moved it aside:

[root@patroni2 ~]# sudo mv /etc/ld.so.conf.d/postgres.conf /tmp/postgres.conf.bak
[root@patroni2 ~]# sudo ldconfig

ldconfig rebuilds the system library cache now based only on standard paths.

Re-Run the Preupgrade Check

After remediation, always re-run the preupgrade command and check the report again. If the fix was successful:

  • The severity should disappear from the REPORT OVERVIEW
  • The issue should no longer appear in leapp-report.txt

This validation loop is important. We are progressively cleaning the system until it is fully compliant for upgrade.

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. Leapp detected loaded kernel drivers which are no longer maintained in RHEL 10.
    2. GRUB2 core will be automatically updated during the upgrade
    3. Failed to read GPG keys from provided key files
    4. Berkeley DB (libdb) has been detected on your system

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       3
    MEDIUM severity reports:     1
    LOW severity reports:        1
    INFO severity reports:       3

Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
    A report has been generated at /var/log/leapp/leapp-report.txt
    A report has been generated at /var/log/leapp/leapp-report.json

Only once the report is clean, or fully understood, should we proceed to the actual upgrade execution.

2. Execute the upgrade

Once all errors and inhibitors are resolved, and high-severity findings have been reviewed or remediated, we are finally ready to perform the actual in-place upgrade. This is the moment where Leapp transitions from analysis mode to execution mode.

About the Repository Warning

During the preupgrade phase, Leapp informed us that codeready-builder-… repositories are not officially supported during the upgrade process and are excluded by default.

This is expected behavior as Leapp only enables a minimal, controlled set of repositories to ensure:

  • Package consistency
  • Dependency resolution stability
  • Predictable upgrade paths

However, in PostgreSQL environments, some packages (extensions, development headers, libraries) may depend on CodeReady Builder. If a repository is truly required during the upgrade, we must explicitly enable it using:

--enablerepo <repoid>

Running the Upgrade

Since I need CodeReady Builder for PostgreSQLdependencies, I ran:

leapp upgrade --target 10.1 --enablerepo codeready-builder-for-rhel-10-x86_64-rpms

What happens when we run this command?

At this stage, Leapp:

  • Resolves and downloads required RHEL 10 packages
  • Builds a temporary upgrade environment
  • Prepares a special upgrade initramfs
  • Modifies the bootloader (GRUB) to boot into the upgrade environment on next reboot

The system is not upgraded immediately. The actual OS transition happens during the next boot.

After the command completes, Leapp generates another report. Just like in the preupgrade phase, verify:

  • Errors: 0
  • Inhibitors: 0

If everything looks clean, we can proceed.

Reboot – The Real Upgrade Begins

# reboot

This is where the real upgrade starts. During boot:

  • The system enters a temporary upgrade environment
  • Packages are replaced
  • Obsolete components are removed
  • Configuration files are migrated
  • The new RHEL 10 kernel is installed

This phase can take several minutes depending on your VM/server resources. My VM doesn’t have many resources and it took me around 30 minutes. Be patient, interrupting this process can leave the system in an inconsistent state.

Verifying the Upgrade

Once the server is back online, confirm the OS version:

[root@patroni2 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="10.1 (Coughlan)"
ID="rhel"
ID_LIKE="centos fedora"
VERSION_ID="10.1"
PLATFORM_ID="platform:el10"
PRETTY_NAME="Red Hat Enterprise Linux 10.1 (Coughlan)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:10.1"
HOME_URL="https://www.redhat.com/"
VENDOR_NAME="Red Hat"
VENDOR_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT=”Red Hat Enterprise Linux 10″
REDHAT_BUGZILLA_PRODUCT_VERSION=10.1
REDHAT_SUPPORT_PRODUCT=”Red Hat Enterprise Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=”10.1″

This confirms that we are now running RHEL 10.1.

III. Post-Upgrade Database Recovery

Once you login to RHEL 10.1, your Postgres binaries in /u01 will fail because libicuuc.so.67 (from RHEL 9) is missing. It can also fail because of other libraries.

14:45:55 postgres@patroni2:/home/postgres/ [test-op-patroni] pgstart
/u01/app/postgres/product/17/db_6/bin/postgres: error while loading shared libraries: libicuuc.so.67: cannot open shared object file: No such file or directory
no data was returned by command ""/u01/app/postgres/product/17/db_6/bin/postgres" -V"
command not found
program "postgres" is needed by pg_ctl but was not found in the same directory as "/u01/app/postgres/product/17/db_6/bin/pg_ctl"

1. Recompile PostgreSQL

Since you installed from source, you must re compile PostgreSQL with the new RHEL 10 system libraries. Here is the command I personally use to build it, with the postgres user:

postgres@patroni2:/home/postgres/ [dummy] MAJOR="17"
postgres@patroni2:/home/postgres/ [dummy] MINOR="6"
postgres@patroni2:/home/postgres/ [dummy] tar axf postgresql-${MAJOR}.${MINOR}.tar.gz
postgres@patroni2:/home/postgres/ [dummy] mkdir build; cd $_
postgres@patroni2:/home/postgres/ [dummy] export PGHOME="/u01/app/postgres/product/${MAJOR}/db_${MINOR}"
postgres@patroni2:/home/postgres/ [dummy] export SEGSIZE=2
postgres@patroni2:/home/postgres/ [dummy] export BLOCKSIZE=8
postgres@patroni2:/home/postgres/ [dummy] meson setup . ../postgresql-${MAJOR}.${MINOR}
postgres@patroni2:/home/postgres/ [dummy] meson configure -Dprefix=${PGHOME}                   -Dbindir=${PGHOME}/bin                   -Ddatadir=${PGHOME}/share                   -Dincludedir=${PGHOME}/include                   -Dlibdir=${PGHOME}/lib                   -Dsysconfdir=${PGHOME}/etc                   -Dpgport=5432                   -Dplperl=enabled                   -Dplpython=enabled                   -Dssl=openssl                   -Dpam=enabled                   -Dldap=enabled                   -Dlibxml=enabled                   -Dlibxslt=enabled                   -Dsegsize=${SEGSIZE}                   -Dblocksize=${BLOCKSIZE}                   -Dllvm=enabled                   -Duuid=ossp                   -Dzstd=enabled                   -Dlz4=enabled                   -Dzstd=enabled                   -Dgssapi=enabled                   -Dsystemd=enabled                   -Dicu=enabled                   -Dsystem_tzdata=/usr/share/zoneinfo                   -Dextra_version=" dbi services build"
postgres@patroni2:/home/postgres/ [dummy] ninja
postgres@patroni2:/home/postgres/ [dummy] ninja install

2. Restore Library Paths

[root@patroni2 ~]# mv /tmp/postgres.conf.bak /etc/ld.so.conf.d/postgres.conf
[root@patroni2 ~]# ldconfig

3. Start & Fix Collation Mismatch

Postgres will now start, but will warn you about Collation Version Mismatches (2.34 vs 2.39).

15:09:55 postgres@patroni2:/home/postgres/build/ [test-op-patroni] pgstart
waiting for server to start.... done
server started
15:10:28 postgres@patroni2:/home/postgres/build/ [test-op-patroni] psql
WARNING:  database "postgres" has a collation version mismatch
DETAIL:  The database was created using collation version 2.34, but the operating system provides version 2.39.
HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE postgres REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
psql (17.6 dbi services build)
Type "help" for help.

Inside Postgres, run for every database:

ALTER DATABASE postgres REFRESH COLLATION VERSION;
ALTER DATABASE

-- Repeat for other DBs if applicable
REINDEX DATABASE postgres;

Your PostgreSQL is now starting properly and your server has been upgraded.

4. In case of a patroni cluster

Since the system Python version has changed after the OS upgrade, your old .local venv is invalid. You must recreate it. Here is how I install patroni using the postgres user:

$ python3 -m venv .local
$ .local/bin/pip3 install –upgrade pip
$ .local/bin/pip3 install –upgrade setuptools
$ .local/bin/pip3 install wheel
$ .local/bin/pip3 install psycopg[binary]
$ .local/bin/pip3 install python-etcd
$ .local/bin/pip3 install patroni
$ .local/bin/patroni version

Check if the cluster sees the member again. If patronictl list is empty, a restart of the service is usually required to re-register with etcd.

10:45:40 postgres@patroni2:/home/postgres/ [test-op-patroni] patronictl list
+ Cluster: test-op-patroni (7565882985963789761) -+-----+------------+-----+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------+------+------+-------+----+-------------+-----+------------+-----+
+--------+------+------+-------+----+-------------+-----+------------+-----+
10:45:48 postgres@patroni2:/home/postgres/ [test-op-patroni] sudo systemctl restart patroni
10:45:55 postgres@patroni2:/home/postgres/ [test-op-patroni] patronictl list
+ Cluster: test-op-patroni (7565882985963789761) --+---------+----+-------------+-----+------------+-----+
| Member                 | Host           | Role   | State   | TL | Receive LSN | Lag | Replay LSN | Lag |
+------------------------+----------------+--------+---------+----+-------------+-----+------------+-----+
| patroni-tst-op-geapg02 | 192.168.56.142 | Leader | running | 11 |             |     |            |     |
+------------------------+----------------+--------+---------+----+-------------+-----+------------+-----+

IV. Conclusion

Upgrading from RHEL 9.6 to 10.1 is a big move. It’s not just a simple update; it’s a total shift in the system’s foundation. Between hardware driver changes and library updates, you really have to pay attention to the details to keep your database running.

RHEL 10.1 is a great, modern platform, but you can’t just click “update” and hope for the best. By planning the upgrade, planning to rebuild your binaries, refreshing your database objects, you can make the jump without the drama. Take the pre-upgrade report seriously, it’s there for a reason!

That said, for a production PostgreSQL cluster, especially one managed with Patroni and etcd, I would not recommend this in-place upgrade approach. Even if Leapp makes the process technically possible, you are still:

  • Modifying the operating system in place
  • Replacing core libraries underneath a running database stack
  • Trusting automated dependency resolution during a major version jump

In production, risk reduction should always be the priority.

Instead, I strongly recommend provisioning new VMs or physical servers, installing RHEL 10.1 from scratch, deploying PostgreSQL, Patroni, and etcd cleanly, rebuilding the cluster from best practices, and then migrating the data from the old environment to the new one using replication or another appropriate method.

Sometimes the safest upgrade… is a new cluster.

Post Views: 118
Thumbnail [60x60]
by
Joan Frey