Your applications still need one stable address to connect to, and they need writes to reach the primary while reads spread across replicas. HAProxy handles that routing, with a floating IP from Keepalived in front of it.

How the tools work together

Three components stands between your application and the database.

Patroni manages replication and failover, and it runs an agent on every PostgreSQL node. Each agent exposes a small REST API (port 8008 by default) that reports that node’s role.

HAProxy accepts client connections and forwards them to the right node. It asks Patroni’s REST API which node is the primary and which are replicas, then sends each connection to a matching node.

Keepalived publishes a virtual IP that floats between your HAProxy hosts using VRRP. Your application connects to the VIP, so one HAProxy host going down doesn’t take the whole entry point with it.

Your application talks to the VIP. Keepalived points the VIP at a live HAProxy. HAProxy forwards the connection to whichever PostgreSQL node Patroni reports as healthy for that role.

The health-check method

HAProxy checks one port and routes to another.

Patroni’s REST API returns an HTTP status that depends on the node’s role:

• GET / returns 200 only on the leader (the primary). A non-leader node returns 503.
• GET /primary is the explicit name for the same leader check.
• GET /replica returns 200 only on a running replica.
• GET /read-only returns 200 on the primary or a replica, any node that can serve a read.

In our case, HAProxy runs its health check against the API port (8008) and reads that status code, then forwards the SQL connection to the database port (5432). A node receives traffic only when its API answers 200 for the role that listener cares about. Point a listener’s check at / and it follows the primary. Point it at /replica and it follows the replicas. Patroni promotes a new leader, the status codes change, and HAProxy moves traffic to match within a couple of health-check cycles.

A first and simple working configuration

A two-node setup with 10.5.5.147 and 10.5.5.148 looks like this. One listener handles writes, the other handles reads.

listen PG1
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

listen PG1_ro
    bind *:5001
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

This example runs PostgreSQL on port 5432 and the Patroni API on 8008, so swap in whatever ports your deployment uses (the defaults are 5432 and 8008).

Line by line:

• bind *:5000 and bind *:5001 are the two addresses your applications connect to. Send writes to 5000 and reads to 5001.
• option httpchk (with no path) on the first listener checks Patroni’s root endpoint. Only the leader answers 200, so HAProxy sends port 5000 traffic to the current primary.
• option httpchk GET /replica on the second listener checks the replica endpoint, so HAProxy sends port 5001 traffic to a replica.
• http-check expect status 200 tells HAProxy that 200 means healthy and anything else means down.
• inter 3s fall 3 rise 2 checks every 3 seconds, marks a server down after 3 failures, and brings it back after 2 successes.
• on-marked-down shutdown-sessions kills existing connections to a server the instant HAProxy marks it down, so clients reconnect and get rerouted instead of hanging on a dead node.
• check port 8008 is the trick in action: health checks hit the Patroni API on 8008 while HAProxy forwards traffic to PostgreSQL on 5432.
• maxconn 100 limit connections per server so you don’t exhaust PostgreSQL’s connection slots.

For a primary plus one or more replicas, this routes writes and reads to the right node and survives a failover.

The failure mode hiding in the read path

Imagine a two-node cluster: one primary, one replica. The replica goes down. Maybe it crashed, maybe Patroni is mid-switchover and no standby exists for a few seconds.

Your read traffic hits port 5001. That listener marks a server up only when GET /replica returns 200, and right now no node is a replica. HAProxy has zero usable servers in the pool, so it refuses the connection. Read queries start failing.

The primary is up the entire time, and it can serve those reads. Your config won’t send them there, because you told the read listener to look for replicas and nothing else. You’ve turned a degraded cluster that could still serve reads into a read outage. You feel this most on small clusters, and each failover passes through a window where the old primary becomes a replica and no standby is available yet. In the worst case, your replica is down, and one of your application is connecting to port 5001, resulting in errors.

The fix: fall back to the primary

Send reads to the primary when the read listener runs out of replicas, instead of dropping them.

listen PG1
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

listen PG1_ro
    bind *:5001
    option httpchk GET /replica
    http-check expect status 200
    use_backend PG1_ro_leader if { nbsrv(PG1_ro) eq 0 }
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

backend PG1_ro_leader
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

This new line carries the whole fix:

use_backend PG1_ro_leader if { nbsrv(PG1_ro) eq 0 }

nbsrv(PG1_ro) counts the usable servers in the PG1_ro pool, which here means the number of available replicas, since those servers pass the check only when GET /replica returns 200. While at least one replica is up, the count stays above zero, the condition is false, and reads stay on the replicas. The moment the last replica drops, nbsrv(PG1_ro) hits zero, the condition fires, and HAProxy diverts reads to the PG1_ro_leader backend.

That backend health-checks with GET /primary, so the only server it counts as up is the current primary. HAProxy sends reads to the primary until a replica returns, then shifts them back to the replica pool once a replica passes its check again.

Three names have to agree for this to work. The backend you define (backend PG1_ro_leader), the backend you route to (use_backend PG1_ro_leader), and the pool you count (nbsrv(PG1_ro)) all reference the real section names. Drop in a stale name from an earlier version and HAProxy either refuses to start or counts the wrong pool.

The /read-only shortcut and what it costs

Patroni offers GET /read-only, which returns 200 on the primary and the replicas alike. Point the read listener there and both the primary and the replicas serve reads, no fallback backend needed.

The cost is read load on the primary even when your replicas are healthy and idle. The fallback approach keeps reads off the primary until the replicas are gone, then leans on it as a safety net. You protect the primary’s write capacity during normal operation and still keep reads alive during a replica outage.

To keep lagging replicas out of the read pool, Patroni accepts a threshold on the replica check, for example GET /replica?lag=10MB, which fails any replica more than 10 MB behind. Pair that with the fallback and HAProxy drops the lagging replicas from rotation while reads still have somewhere to go.

Keepalived: removing HAProxy as a single point of failure

One HAProxy host fronting the cluster moves the single point of failure up a layer. Run HAProxy on two hosts and let Keepalived float a virtual IP between them with VRRP. Your application connects to the VIP, and whichever HAProxy holds it answers.

A minimal keepalived.conf on the primary HAProxy host:

vrrp_script chk_haproxy {
    script "killall -0 haproxy"   # succeeds while the haproxy process is alive
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass changeme
    }
    virtual_ipaddress {
        10.0.0.1
    }
    track_script {
        chk_haproxy
    }
}

The second HAProxy host runs the same file with state BACKUP and a lower priority (100). Both advertise over VRRP, and the higher priority holds the VIP. chk_haproxy runs every two seconds. If HAProxy dies on the active host, its priority drops and the backup takes the VIP, so an HAProxy crash on one host no longer takes the entry point down with it.

Point your applications at 10.0.0.1:5000 for writes and 10.5.5.100:5001 for reads. Your applications never see which physical HAProxy does the work.

Summary

Patroni keeps the cluster healthy and picks the leader. HAProxy turns Patroni’s REST API into routing, sending writes to the primary and reads to the replicas by health-checking the API port while forwarding to the database port. The naive read-only listener drops reads when the last replica goes down, even though the primary could serve them. Adding use_backend ... if { nbsrv(...) eq 0 } with a primary-checking backend closes that gap, and a lag threshold on the replica check keeps stale standbys out of rotation. Keepalived puts a floating VIP in front of two HAProxy instances so the proxy layer survives a host failure too.

Writes reach the primary and reads spread across the replicas. Reads stay up as long as one node in the cluster is alive.

Let me know if you find any improvements to this configuration

L’article Highly Available, Load-Balanced PostgreSQL with Patroni, HAProxy, and Keepalived est apparu en premier sur dbi Blog.

In this guide, I walk through a real-world in-place upgrade using Leapp, covering preparation, resolving high-severity warnings, executing the upgrade, recompiling PostgreSQL, fixing collation mismatches, and restoring Patroni.

I. Preparation

Before the upgrade, you must ensure the current OS is healthy and fully patched.

1. Pause High Availability

Prevent Patroni from triggering a failover during the reboot cycles. If you are using a single PostgreSQL cluster, stop it by stopping the service or by using pg_ctl stop.

patronictl -c /etc/patroni/patroni.yml pause
systemctl stop patroni

2. Fix Subscription & Perform Full Update

I’m using an old VM for this blog, and If just like me, you see 403 Forbidden errors on repositories like codeready-builder, refresh your registration:

[root@patroni2 ~]# sudo dnf update -y
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs)                                                                                                     761  B/s | 480  B     00:00
Errors during downloading metadata for repository 'codeready-builder-for-rhel-9-x86_64-rpms':
  - Status code: 403 for https://cdn.redhat.com/content/dist/rhel9/9/x86_64/codeready-builder/os/repodata/repomd.xml (IP: 23.206.57.92)
Error: Failed to download metadata for repo 'codeready-builder-for-rhel-9-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[root@patroni2 ~]# sudo subscription-manager clean
[root@patroni2 ~]# sudo subscription-manager register --force
[root@patroni2 ~]# sudo subscription-manager attach --auto
[root@patroni2 ~]# sudo subscription-manager refresh

[root@patroni2 ~]# sudo dnf update -y
...
Complete!

[root@patroni2 ~]# reboot

II. The Leapp Upgrade to 10.1

1. Install & Analyze

In this first phase, we are preparing the system for a major in-place upgrade using Leapp, the official upgrade framework for Red Hat–based distributions. When we install the package:

[root@patroni2 ~]# dnf install leapp-upgrade -y
...
Installed:
  leapp-0.20.0-1.el9.noarch                leapp-deps-0.20.0-1.el9.noarch          leapp-upgrade-el9toel10-0.23.0-1.el9.noarch       leapp-upgrade-el9toel10-deps-0.23.0-1.el9.noarch
  libdb-utils-5.3.28-57.el9_6.x86_64       python3-leapp-0.20.0-1.el9.noarch       systemd-container-252-55.el9_7.7.x86_64

Complete!

When running leapp preupgrade –target 10.1, we are not performing the upgrade. Instead, Leapp performs a full system audit to determine if the server is ready for RHEL 10.1. It checks:

• Installed packages and their compatibility
• Deprecated or removed libraries
• Kernel drivers that will not exist in RHEL 10
• Bootloader configuration (GRUB2)
• GPG key validity
• Custom system-level modifications (like dynamic linker changes)
• …

Think of this step as a dry-run with intelligence.

[root@patroni2 ~]# sudo leapp preupgrade --target 10.1

...

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. GRUB2 core will be automatically updated during the upgrade
    2. Detected customized configuration for dynamic linker.
    3. Leapp detected loaded kernel drivers which are no longer maintained in RHEL 10.
    4. Failed to read GPG keys from provided key files
    5. Berkeley DB (libdb) has been detected on your system

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       4
    MEDIUM severity reports:     1
    LOW severity reports:        1
    INFO severity reports:       3

Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
    A report has been generated at /var/log/leapp/leapp-report.txt
    A report has been generated at /var/log/leapp/leapp-report.json

After running the pre-upgrade analysis, the next step is to carefully review:

/var/log/leapp/leapp-report.txt

What we are looking for first is simple:

• Errors: 0
• Inhibitors: 0

If an Inhibitor is present, the upgrade will be blocked entirely.
In my case, there were no blockers, but I did have several high severity warnings.

High severity does not mean the upgrade will fail.
It means: This could break something, review it carefully.

Let’s look at one concrete example from my system.

2. High Severity Example – Dynamic Linker Customization

Leapp detected that my system had a custom dynamic linker configuration:

Risk Factor: high

Title: Detected customized configuration for dynamic linker.

Summary: Custom configurations to the dynamic linker could potentially impact the upgrade in a negative way. The custom configuration includes modifications to /etc/ld.so.conf, custom or modified drop in config files in the /etc/ld.so.conf.d directory and additional entries in the LD_LIBRARY_PATH or LD_PRELOAD variables. These modifications configure the dynamic linker to use different libraries that might not be provided by Red Hat products or might not be present during the whole upgrade process. The following custom configurations were detected by leapp:

- The following drop in config files were marked as custom:

    - /etc/ld.so.conf.d/postgres.conf

Remediation: [hint] Remove or revert the custom dynamic linker configurations and apply the changes using the ldconfig command. In case of possible active software collections we suggest disabling them persistently.

Key: cc9bd972af70b7a27f66a37b11a00dcfcb73b1bc

----------------------------------------

What does this actually mean?

The dynamic linker (ld.so) is responsible for loading shared libraries at runtime.

By modifying:

• /etc/ld.so.conf
• files in /etc/ld.so.conf.d/
• LD_LIBRARY_PATH
• LD_PRELOAD

we are telling the system to load non-standard or custom libraries. In PostgreSQL environments (especially with custom builds or extensions), this is common practice. However, during a major OS upgrade, these custom paths might:

• Point to libraries that do not exist in RHEL 10
• Override new system libraries
• Break dependency resolution mid-upgrade

Leapp flags this because it cannot guarantee consistency during the transition phase. In my case, it shouldn’t be an issue, because inside postgres.conf, I only have a path aiming to the lib directories of my PostgreSQL installation, which will not change, but we will still see how to prevent an error.

Understanding the Remediation

The report clearly suggests:

Remove or revert the custom dynamic linker configurations and apply the changes using the ldconfig command.

In my case, the configuration was related to PostgreSQL, so temporarily removing it is safe for the upgrade preparation phase. Instead of deleting it permanently, I moved it aside:

[root@patroni2 ~]# sudo mv /etc/ld.so.conf.d/postgres.conf /tmp/postgres.conf.bak
[root@patroni2 ~]# sudo ldconfig

ldconfig rebuilds the system library cache now based only on standard paths.

Re-Run the Preupgrade Check

After remediation, always re-run the preupgrade command and check the report again. If the fix was successful:

• The severity should disappear from the REPORT OVERVIEW
• The issue should no longer appear in leapp-report.txt

This validation loop is important. We are progressively cleaning the system until it is fully compliant for upgrade.

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. Leapp detected loaded kernel drivers which are no longer maintained in RHEL 10.
    2. GRUB2 core will be automatically updated during the upgrade
    3. Failed to read GPG keys from provided key files
    4. Berkeley DB (libdb) has been detected on your system

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       3
    MEDIUM severity reports:     1
    LOW severity reports:        1
    INFO severity reports:       3

Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
    A report has been generated at /var/log/leapp/leapp-report.txt
    A report has been generated at /var/log/leapp/leapp-report.json

Only once the report is clean, or fully understood, should we proceed to the actual upgrade execution.

2. Execute the upgrade

Once all errors and inhibitors are resolved, and high-severity findings have been reviewed or remediated, we are finally ready to perform the actual in-place upgrade. This is the moment where Leapp transitions from analysis mode to execution mode.

About the Repository Warning

During the preupgrade phase, Leapp informed us that codeready-builder-… repositories are not officially supported during the upgrade process and are excluded by default.

This is expected behavior as Leapp only enables a minimal, controlled set of repositories to ensure:

• Package consistency
• Dependency resolution stability
• Predictable upgrade paths

However, in PostgreSQL environments, some packages (extensions, development headers, libraries) may depend on CodeReady Builder. If a repository is truly required during the upgrade, we must explicitly enable it using:

--enablerepo <repoid>

Running the Upgrade

Since I need CodeReady Builder for PostgreSQLdependencies, I ran:

leapp upgrade --target 10.1 --enablerepo codeready-builder-for-rhel-10-x86_64-rpms

What happens when we run this command?

At this stage, Leapp:

• Resolves and downloads required RHEL 10 packages
• Builds a temporary upgrade environment
• Prepares a special upgrade initramfs
• Modifies the bootloader (GRUB) to boot into the upgrade environment on next reboot

The system is not upgraded immediately. The actual OS transition happens during the next boot.

After the command completes, Leapp generates another report. Just like in the preupgrade phase, verify:

• Errors: 0
• Inhibitors: 0

If everything looks clean, we can proceed.

Reboot – The Real Upgrade Begins

# reboot

This is where the real upgrade starts. During boot:

• The system enters a temporary upgrade environment
• Packages are replaced
• Obsolete components are removed
• Configuration files are migrated
• The new RHEL 10 kernel is installed

This phase can take several minutes depending on your VM/server resources. My VM doesn’t have many resources and it took me around 30 minutes. Be patient, interrupting this process can leave the system in an inconsistent state.

Verifying the Upgrade

Once the server is back online, confirm the OS version:

[root@patroni2 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="10.1 (Coughlan)"
ID="rhel"
ID_LIKE="centos fedora"
VERSION_ID="10.1"
PLATFORM_ID="platform:el10"
PRETTY_NAME="Red Hat Enterprise Linux 10.1 (Coughlan)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:10.1"
HOME_URL="https://www.redhat.com/"
VENDOR_NAME="Red Hat"
VENDOR_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT=”Red Hat Enterprise Linux 10″
REDHAT_BUGZILLA_PRODUCT_VERSION=10.1
REDHAT_SUPPORT_PRODUCT=”Red Hat Enterprise Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=”10.1″

This confirms that we are now running RHEL 10.1.

III. Post-Upgrade Database Recovery

Once you login to RHEL 10.1, your Postgres binaries in /u01 will fail because libicuuc.so.67 (from RHEL 9) is missing. It can also fail because of other libraries.

14:45:55 postgres@patroni2:/home/postgres/ [test-op-patroni] pgstart
/u01/app/postgres/product/17/db_6/bin/postgres: error while loading shared libraries: libicuuc.so.67: cannot open shared object file: No such file or directory
no data was returned by command ""/u01/app/postgres/product/17/db_6/bin/postgres" -V"
command not found
program "postgres" is needed by pg_ctl but was not found in the same directory as "/u01/app/postgres/product/17/db_6/bin/pg_ctl"

1. Recompile PostgreSQL

Since you installed from source, you must re compile PostgreSQL with the new RHEL 10 system libraries. Here is the command I personally use to build it, with the postgres user:

postgres@patroni2:/home/postgres/ [dummy] MAJOR="17"
postgres@patroni2:/home/postgres/ [dummy] MINOR="6"
postgres@patroni2:/home/postgres/ [dummy] tar axf postgresql-${MAJOR}.${MINOR}.tar.gz
postgres@patroni2:/home/postgres/ [dummy] mkdir build; cd $_
postgres@patroni2:/home/postgres/ [dummy] export PGHOME="/u01/app/postgres/product/${MAJOR}/db_${MINOR}"
postgres@patroni2:/home/postgres/ [dummy] export SEGSIZE=2
postgres@patroni2:/home/postgres/ [dummy] export BLOCKSIZE=8
postgres@patroni2:/home/postgres/ [dummy] meson setup . ../postgresql-${MAJOR}.${MINOR}
postgres@patroni2:/home/postgres/ [dummy] meson configure -Dprefix=${PGHOME}                   -Dbindir=${PGHOME}/bin                   -Ddatadir=${PGHOME}/share                   -Dincludedir=${PGHOME}/include                   -Dlibdir=${PGHOME}/lib                   -Dsysconfdir=${PGHOME}/etc                   -Dpgport=5432                   -Dplperl=enabled                   -Dplpython=enabled                   -Dssl=openssl                   -Dpam=enabled                   -Dldap=enabled                   -Dlibxml=enabled                   -Dlibxslt=enabled                   -Dsegsize=${SEGSIZE}                   -Dblocksize=${BLOCKSIZE}                   -Dllvm=enabled                   -Duuid=ossp                   -Dzstd=enabled                   -Dlz4=enabled                   -Dzstd=enabled                   -Dgssapi=enabled                   -Dsystemd=enabled                   -Dicu=enabled                   -Dsystem_tzdata=/usr/share/zoneinfo                   -Dextra_version=" dbi services build"
postgres@patroni2:/home/postgres/ [dummy] ninja
postgres@patroni2:/home/postgres/ [dummy] ninja install

2. Restore Library Paths

[root@patroni2 ~]# mv /tmp/postgres.conf.bak /etc/ld.so.conf.d/postgres.conf
[root@patroni2 ~]# ldconfig

3. Start & Fix Collation Mismatch

Postgres will now start, but will warn you about Collation Version Mismatches (2.34 vs 2.39).

15:09:55 postgres@patroni2:/home/postgres/build/ [test-op-patroni] pgstart
waiting for server to start.... done
server started
15:10:28 postgres@patroni2:/home/postgres/build/ [test-op-patroni] psql
WARNING:  database "postgres" has a collation version mismatch
DETAIL:  The database was created using collation version 2.34, but the operating system provides version 2.39.
HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE postgres REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
psql (17.6 dbi services build)
Type "help" for help.

Inside Postgres, run for every database:

ALTER DATABASE postgres REFRESH COLLATION VERSION;
ALTER DATABASE

-- Repeat for other DBs if applicable
REINDEX DATABASE postgres;

Your PostgreSQL is now starting properly and your server has been upgraded.

4. In case of a patroni cluster

Since the system Python version has changed after the OS upgrade, your old .local venv is invalid. You must recreate it. Here is how I install patroni using the postgres user:

$ python3 -m venv .local
$ .local/bin/pip3 install –upgrade pip
$ .local/bin/pip3 install –upgrade setuptools
$ .local/bin/pip3 install wheel
$ .local/bin/pip3 install psycopg[binary]
$ .local/bin/pip3 install python-etcd
$ .local/bin/pip3 install patroni
$ .local/bin/patroni version

Check if the cluster sees the member again. If patronictl list is empty, a restart of the service is usually required to re-register with etcd.

10:45:40 postgres@patroni2:/home/postgres/ [test-op-patroni] patronictl list
+ Cluster: test-op-patroni (7565882985963789761) -+-----+------------+-----+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------+------+------+-------+----+-------------+-----+------------+-----+
+--------+------+------+-------+----+-------------+-----+------------+-----+
10:45:48 postgres@patroni2:/home/postgres/ [test-op-patroni] sudo systemctl restart patroni
10:45:55 postgres@patroni2:/home/postgres/ [test-op-patroni] patronictl list
+ Cluster: test-op-patroni (7565882985963789761) --+---------+----+-------------+-----+------------+-----+
| Member                 | Host           | Role   | State   | TL | Receive LSN | Lag | Replay LSN | Lag |
+------------------------+----------------+--------+---------+----+-------------+-----+------------+-----+
| patroni-tst-op-geapg02 | 192.168.56.142 | Leader | running | 11 |             |     |            |     |
+------------------------+----------------+--------+---------+----+-------------+-----+------------+-----+

IV. Conclusion

Upgrading from RHEL 9.6 to 10.1 is a big move. It’s not just a simple update; it’s a total shift in the system’s foundation. Between hardware driver changes and library updates, you really have to pay attention to the details to keep your database running.

RHEL 10.1 is a great, modern platform, but you can’t just click “update” and hope for the best. By planning the upgrade, planning to rebuild your binaries, refreshing your database objects, you can make the jump without the drama. Take the pre-upgrade report seriously, it’s there for a reason!

That said, for a production PostgreSQL cluster, especially one managed with Patroni and etcd, I would not recommend this in-place upgrade approach. Even if Leapp makes the process technically possible, you are still:

• Modifying the operating system in place
• Replacing core libraries underneath a running database stack
• Trusting automated dependency resolution during a major version jump

In production, risk reduction should always be the priority.

Instead, I strongly recommend provisioning new VMs or physical servers, installing RHEL 10.1 from scratch, deploying PostgreSQL, Patroni, and etcd cleanly, rebuilding the cluster from best practices, and then migrating the data from the old environment to the new one using replication or another appropriate method.

Sometimes the safest upgrade… is a new cluster.

L’article Upgrade RHEL from 9.6 to 10.1 (when running PostgreSQL/Patroni) est apparu en premier sur dbi Blog.

Monitoring a PostgreSQL environment involves much more than simply checking whether the database is running. In a modern high availability architecture, multiple components work together to keep the service available and stable.

In this article, we will explain how to monitor a PostgreSQL cluster environment with Zabbix:

• PostgreSQL
• Patroni
• HAProxy
• ETCD
• Keepalived

The goal of this blog is to provide a simple and clear overview that can be understood by both beginners and experienced administrators.

PostgreSQL High Availability Architecture

A typical PostgreSQL high availability environment often contains several components:

In a PostgreSQL HA environment, monitoring only the database is not enough because every component directly impacts cluster stability and application availability. Moreover, a failures in components like HAProxy, ETCD, or Keepalived can directly impact application availability and cluster stability. Comprehensive monitoring ensures the entire ecosystem remains healthy and operational.

Monitoring PostgreSQL with Zabbix

Zabbix already provides an official PostgreSQL template.

This template allows administrators to monitor:

• Database availability
• Connections
• Transactions
• Cache hit ratio
• Replication status
• Locks
• Deadlocks
• Database size
• Query statistics
• WAL activity
• Performance metrics

Therefore, using the official template is usually the fastest and easiest way to start monitoring PostgreSQL.

ETCD

Zabbix already provides an official template for ETCD monitoring.

The template uses HTTP Agent items to collect metrics directly from the /metrics endpoint exposed by ETCD.

In addition, the template works without external scripts and uses Prometheus-style metrics collection.

The official template already includes many useful triggers and metrics for monitoring ETCD health, performance, and cluster activity.

HAProxy

HAProxy plays a critical role in a PostgreSQL high availability environment, ensuring that client connections are automatically routed to the correct PostgreSQL node and maintaining application connectivity during failovers.

Zabbix already provides an official HAProxy monitoring template that can be used as a base for monitoring and customization: HAProxy by HTTP

The number of active servers in HAProxy is also monitored by the template. However, the template does not create a trigger by default for this metric. This means that a backend could lose one or more active servers without generating any alert in Zabbix. For this reason, it is highly recommended to create custom triggers on this item to immediately detect backend availability issues and avoid unnoticed service degradation.

To implement this properly, a new macro should be created in the template to define the minimum expected number of active servers. Then, a custom trigger must be added on the item prototype available in the Low-Level Discovery (LLD) section of the template. Consequently, Zabbix can automatically monitor and alert on all discovered HAProxy backends.

Patroni

Zabbix does not directly include Patroni monitoring in the standard PostgreSQL templates.

One approach is to use the Zabbix Systemd template, which allows monitoring of services running on the server. In this case, it is simply necessary to filter on the patroni service name to verify that the service is active.

Another approach is to create a custom item in a template using a Zabbix agent item with the following key:

net.tcp.service["{$HAPROXY.STATS.SCHEME}","{$HAPROXY.STATS.HOST}","{$HAPROXY.STATS.PORT.PATRONI}"]

This item performs a direct HTTP TCP connection test on port 5000, which is commonly used by the Patroni API. This configuration allows Zabbix to verify that the Patroni service responds correctly and remains reachable.

Additionally, combining both monitoring methods significantly improves monitoring reliability.

Monitoring Keepalived

Keepalived monitoring is not included by default in the standard Zabbix templates.
To properly monitor the VIP management layer, administrators should create custom items whether the Virtual IP (VIP) is correctly present on one of the cluster nodes.

A common approach is to create three different items:

• Two items using the net.tcp.service key to test TCP connectivity for:
  • the READ-ONLY access
  • the READ-WRITE access
• One additional item using a system.run command to directly verify the VIP status on the server.

Together, these checks help ensure that:

• the VIP is correctly assigned
• the PostgreSQL services are reachable
• the failover mechanism is working properly

Finally, administrators should create the appropriate triggers for these items in order to immediately detect VIP failover or connectivity issues.

Conclusion

In conclusion, monitoring PostgreSQL alone is not enough in a high availability environment.

A complete monitoring strategy must include:

• PostgreSQL
• Patroni
• HAProxy
• ETCD
• Keepalived

Using Zabbix with both official and custom templates provides a centralized and efficient monitoring solution.

The PostgreSQL template available in Zabbix already offers excellent database monitoring capabilities.

Additional templates for ETCD and HAProxy help extend monitoring to the full HA architecture and improve visibility across the entire platform.

As a result, this approach helps administrators detect failures earlier, improve troubleshooting, and ensure better service availability.

If you need you can try other blogs regarding Zabbix or postgreSQL here: dbi Blog

L’article Zabbix – Monitoring full cluster Patroni/PostgreSQL est apparu en premier sur dbi Blog.

In this article, we will see how to monitor Oracle backups using Zabbix and the Oracle plugin available with Zabbix Agent 2.

This guide assumes that:

• ARCHIVELOG mode is already enabled,
• RMAN backups are already configured,
• and Zabbix Agent 2 is already installed on the Oracle server.

Setup Zabbix agent 2

By default, the Oracle template provided with Zabbix does not include items dedicated to backup monitoring.
To achieve this, we will use the Oracle plugin feature called CustomQueries. You can find the documentation for this plugin on the official web-site.

The approach is straightforward:

• create custom SQL queries,
• store them locally on the database server,
• and let Zabbix execute them periodically

First, create a directory that will contain your monitoring SQL scripts:

mkdir -p /etc/zabbix/oracle/sql 

chmod 755 -R /etc/zabbix/oracle/sql

Create a SQL script that returns the timestamp of the latest level 0 backup.

vi /etc/zabbix/oracle/sql/last_inc0.sql

SELECT
    CAST(((CAST(BS.COMPLETION_TIME AS DATE) - DATE '1970-01-01') * 86400) AS NUMERIC) AS LAST_INC0
FROM
    V$BACKUP_SET BS
WHERE
    BS.INCREMENTAL_LEVEL = 0
ORDER BY
    BS.COMPLETION_TIME DESC
FETCH FIRST 1 ROWS ONLY;

This query retrieves the completion time of the latest RMAN level 0 backup. Then, it converts the value into Unix epoch format, which makes preprocessing and trigger calculations much easier inside Zabbix.

You can easily adapt this query to monitor:

• archive log backups,
• level 1 incremental backups,
• controlfile backups,
• or even backup duration.

Configure the Oracle Plugin

Now edit the Oracle plugin configuration file:

vi /etc/zabbix/zabbix_agent2.d/plugins.d/oracle.conf

Plugins.Oracle.CustomQueriesPath=/etc/zabbix/oracle/sql/

With this configuration, the Oracle plugin knows exactly where to find your custom SQL scripts.

Afterward, restart the Zabbix Agent 2 service:

systemctl restart zabbix-agent2

Setup Zabbix template

In your Oracle template, create a new Zabbix agent item using the following key:

oracle.custom.query[“{$ORACLE.CONNSTRING}”,”{$ORACLE.USER}”,”{$ORACLE.PASSWORD}”,”{$ORACLE.SERVICE}”,last_inc0]

Make sure that:

• the query name matches your SQL file name (last_inc0.sql),
• and the preprocessing steps are correctly configured.

Once the item becomes active, Zabbix immediately starts collecting the timestamp of the latest level 0 backup.

• And now you can see your last backup time. Then you just have to create a trigger with this item and you have finished !

Conclusion

By using custom Oracle queries with Zabbix Agent 2, you can quickly extend the default Oracle monitoring capabilities with very little configuration effort.

More importantly, this approach helps you detect backup issues early before the FRA fills up and impacts production systems.

Because the solution remains fully customizable, you can easily extend it later to monitor additional RMAN metrics that match your operational requirements.

You can find other blog regarding Zabbix with this link.

L’article Zabbix – Oracle backups monitoring est apparu en premier sur dbi Blog.

The role of the API is deliberately limited: it receives a snapshot request, checks that the requested zvol is authorized, and then runs the zfs snapshot command on the Proxmox side. An allowlist is used to restrict the ZFS volumes that can be accessed. This prevents a REST call from being able to manipulate any dataset on the server.

With this approach, we can reproduce a behavior close to what an enterprise storage array provides, but using Proxmox and ZFS. It is important to note that Proxmox does not natively provide the same level of integration as Pure Storage for SQL Server snapshots. Pure Storage provides dedicated mechanisms and integrations. In our case, we need to build a specific orchestration layer. The REST API therefore acts as an adapter between SQL Server, which drives the snapshot backup workflow, and ZFS, which actually performs the storage-level snapshot.

Architecture

Here is a global overview of the architecture:

• SQL Server freezes the database I/Os
• SQL Server 2025 calls the internal REST API
• The REST API validates the request and checks the zvol allowlist
• The API triggers the ZFS snapshot on Proxmox
• The API returns the snapshot information to SQL Server
• SQL Server creates the metadata-only backup
• The database I/Os are released

REST API implementation

Under Proxmox, we install the required packages:

apt update
apt install -y python3-venv sudo openssl

We create a dedicated user:

useradd --system \
  --home /opt/sql-zfs-api \
  --shell /usr/sbin/nologin \
  sqlsnap

We create the following folders:

mkdir -p /opt/sql-zfs-api
mkdir -p /etc/sql-zfs-api

We declare the authorized zvol :

cat >/etc/sql-zfs-api/allowed-zvols <<'EOF'
sqlpool/pve/vm-302-disk-0
EOF

We create a root-only allowlist:

chown root:root /etc/sql-zfs-api/allowed-zvols
chmod 600 /etc/sql-zfs-api/allowed-zvols

Then we create the secured ZFS helper. This script is executed as root through sudo, but it rejects any dataset that is not defined in the allowlist.

cat >/usr/local/sbin/sql-zfs-helper <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

ALLOW_FILE="/etc/sql-zfs-api/allowed-zvols"
LOCK_FILE="/run/sql-zfs-helper.lock"

die() {
  echo "$*" >&2
  exit 1
}

exec 9>"$LOCK_FILE"
flock -n 9 || die "another snapshot operation is already running"

[[ -r "$ALLOW_FILE" ]] || die "allowlist not readable: $ALLOW_FILE"

mapfile -t ALLOWED_DATASETS < <(grep -Ev '^\s*(#|$)' "$ALLOW_FILE")

is_allowed() {
  local ds="$1"
  local allowed
  for allowed in "${ALLOWED_DATASETS[@]}"; do
    [[ "$ds" == "$allowed" ]] && return 0
  done
  return 1
}

valid_snapname() {
  [[ "$1" =~ ^[A-Za-z0-9_.:-]{1,120}$ ]]
}

ACTION="${1:-}"
shift || true

case "$ACTION" in
  snapshot)
    SNAPNAME="${1:-}"
    shift || true

    valid_snapname "$SNAPNAME" || die "invalid snapshot name: $SNAPNAME"
    [[ "$#" -ge 1 ]] || die "no zvol specified"
    [[ "$#" -le 8 ]] || die "too many zvols"

    SNAPSHOTS=()

    for DS in "$@"; do
      is_allowed "$DS" || die "dataset not allowed: $DS"
      /sbin/zfs list -H -t volume -o name "$DS" >/dev/null 2>&1 || die "zvol not found: $DS"

      FULLSNAP="${DS}@${SNAPNAME}"

      if /sbin/zfs list -H -t snapshot -o name "$FULLSNAP" >/dev/null 2>&1; then
        die "snapshot already exists: $FULLSNAP"
      fi

      SNAPSHOTS+=("$FULLSNAP")
    done

    /sbin/zfs snapshot "${SNAPSHOTS[@]}"
    /sbin/zfs hold sqlsnap "${SNAPSHOTS[@]}"

    printf '{"status":"ok","snapshots":['
    SEP=""
    for S in "${SNAPSHOTS[@]}"; do
      printf '%s"%s"' "$SEP" "$S"
      SEP=","
    done
    printf ']}\n'
    ;;

  list)
    /sbin/zfs list -H -t snapshot -o name -r sqlpool | grep '@sql_' || true
    ;;

  *)
    die "usage: sql-zfs-helper snapshot SNAPNAME ZVOL [ZVOL...]"
    ;;
esac
EOF

chown root:root /usr/local/sbin/sql-zfs-helper
chmod 750 /usr/local/sbin/sql-zfs-helper

We only allow the helper through sudo:

cat >/etc/sudoers.d/sql-zfs-helper <<'EOF'
sqlsnap ALL=(root) NOPASSWD: /usr/local/sbin/sql-zfs-helper *
EOF

chmod 440 /etc/sudoers.d/sql-zfs-helper
visudo -cf /etc/sudoers.d/sql-zfs-helper

We install the FastAPI API:

python3 -m venv /opt/sql-zfs-api/venv
/opt/sql-zfs-api/venv/bin/pip install fastapi "uvicorn[standard]"

We create the application file:

cat >/opt/sql-zfs-api/app.py <<'EOF'
import os
import re
import json
import socket
import secrets
import subprocess
from datetime import datetime, timezone
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field

API_KEY = os.environ.get("SQL_ZFS_API_KEY", "")
ALLOW_FILE = "/etc/sql-zfs-api/allowed-zvols"
SNAP_RE = re.compile(r"^[A-Za-z0-9_.:-]{1,120}$")

app = FastAPI(title="SQL ZFS Snapshot API", version="1.0.0")


class SnapshotRequest(BaseModel):
    database: str = Field(..., min_length=1, max_length=128)
    vmid: int = 302
    snapname: str = Field(..., min_length=1, max_length=120)
    zvols: list[str] = Field(..., min_length=1, max_length=8)


def load_allowed_zvols() -> set[str]:
    with open(ALLOW_FILE, "r", encoding="utf-8") as f:
        return {
            line.strip()
            for line in f
            if line.strip() and not line.strip().startswith("#")
        }


def check_api_key(x_sqlsnap_key: str | None) -> None:
    if not API_KEY:
        raise HTTPException(status_code=500, detail="API key not configured")

    if not x_sqlsnap_key:
        raise HTTPException(status_code=401, detail="missing API key")

    if not secrets.compare_digest(x_sqlsnap_key, API_KEY):
        raise HTTPException(status_code=403, detail="invalid API key")


@app.get("/health")
def health():
    return {
        "status": "ok",
        "host": socket.gethostname(),
        "utc": datetime.now(timezone.utc).isoformat(),
    }


@app.post("/v1/sql-zfs/snapshot")
def create_snapshot(
    req: SnapshotRequest,
    x_sqlsnap_key: str | None = Header(default=None, alias="x-sqlsnap-key"),
):
    check_api_key(x_sqlsnap_key)

    if not SNAP_RE.fullmatch(req.snapname):
        raise HTTPException(status_code=400, detail="invalid snapname")

    allowed = load_allowed_zvols()

    for zvol in req.zvols:
        if zvol not in allowed:
            raise HTTPException(status_code=403, detail=f"zvol not allowed: {zvol}")

    cmd = [
        "sudo",
        "/usr/local/sbin/sql-zfs-helper",
        "snapshot",
        req.snapname,
        *req.zvols,
    ]

    try:
        completed = subprocess.run(
            cmd,
            text=True,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            timeout=30,
            check=False,
        )
    except subprocess.TimeoutExpired:
        raise HTTPException(status_code=504, detail="zfs snapshot timeout")

    if completed.returncode != 0:
        raise HTTPException(
            status_code=500,
            detail={
                "error": completed.stderr.strip(),
                "stdout": completed.stdout.strip(),
            },
        )

    snapshots = [f"{zvol}@{req.snapname}" for zvol in req.zvols]

    return {
        "status": "ok",
        "database": req.database,
        "vmid": req.vmid,
        "snapname": req.snapname,
        "snapshots": snapshots,
        "media_description": "zfs|" + socket.gethostname() + "|" + ";".join(snapshots),
    }
EOF

chown -R root:root /opt/sql-zfs-api
chmod 755 /opt/sql-zfs-api
chmod 644 /opt/sql-zfs-api/app.py

We configure and generate the key:

APIKEY="$(openssl rand -hex 32)"
echo "$APIKEY"

We create the environment file:

cat >/etc/sql-zfs-api/sql-zfs-api.env <<EOF
SQL_ZFS_API_KEY=$APIKEY
EOF

chown root:root /etc/sql-zfs-api/sql-zfs-api.env
chmod 600 /etc/sql-zfs-api/sql-zfs-api.env

We need to save the generated key.

Next, we enable HTTPS. SQL Server sp_invoke_external_rest_endpoint calls HTTPS endpoints, and the documentation specifies that only HTTPS endpoints with TLS are supported.

openssl req -x509 -newkey rsa:4096 -sha256 -days 360 -nodes \
  -keyout /etc/sql-zfs-api/tls.key \
  -out /etc/sql-zfs-api/tls.crt \
  -subj "/CN=promox1" \
  -addext "subjectAltName=DNS:promox1,IP:192.168.1.110"

chown root:sqlsnap /etc/sql-zfs-api/tls.key /etc/sql-zfs-api/tls.crt
chmod 640 /etc/sql-zfs-api/tls.key
chmod 644 /etc/sql-zfs-api/tls.crt

The /etc/sql-zfs-api/tls.crt certificate must be imported into the Windows trusted root certification authorities on the SQL Server side. Otherwise, the HTTPS call may fail.

We create the systemd service:

cat >/etc/systemd/system/sql-zfs-api.service <<'EOF'
[Unit]
Description=SQL Server to ZFS Snapshot API
After=network-online.target
Wants=network-online.target

[Service]
User=sqlsnap
Group=sqlsnap
WorkingDirectory=/opt/sql-zfs-api
EnvironmentFile=/etc/sql-zfs-api/sql-zfs-api.env
ExecStart=/opt/sql-zfs-api/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8443 --ssl-keyfile /etc/sql-zfs-api/tls.key --ssl-certfile /etc/sql-zfs-api/tls.crt
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now sql-zfs-api
systemctl status sql-zfs-api

We check the status of our API:

It is possible to call the API in PowerShell using Invoke-RestMethod with PowerShell 7:

$headers = @{
"Content-Type"  = "application/json"
"x-sqlsnap-key" = "MyKey"
}

$body = @{
database = "StackOverflow"
vmid     = 302
snapname = "StackOverflow_test010"
zvols    = @("sqlpool/pve/vm-302-disk-0")
} | ConvertTo-Json -Depth 5

Invoke-RestMethod `
-Uri "https://192.168.1.110:8443/v1/sql-zfs/snapshot" `
-Method Post `
-Headers $headers `
-Body $body `
-ContentType "application/json" `
-SkipCertificateCheck

This gives:

Test from SQL Server

A certificate was generated on Proxmox and it needs to be imported on the SQL Server host. In my case, it was located here:

I then imported it on Windows Server:

For testing purposes, I created something simple. On the SQL Server side, we can create a database that will be used to store our future stored procedure. This procedure will allow us to interact with the API. In my case, I created a database called dbi_tools:

This database will contain a credential. In our case, the DATABASE SCOPED CREDENTIAL is used to securely store the authentication information required to call the REST API from SQL Server. This allows us, for example, to protect the API key:

USE [dbi_tools]
GO

IF NOT EXISTS (
    SELECT 1
    FROM sys.symmetric_keys
    WHERE name = '##MS_DatabaseMasterKey##'
)
BEGIN
    CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'MyStrongPassword_%99';
END
GO

CREATE DATABASE SCOPED CREDENTIAL [https://192.168.1.110:8443/v1/sql-zfs/snapshot]
WITH
    IDENTITY = 'HTTPEndpointHeaders',
    SECRET = '{"x-sqlsnap-key":"MyAPIKey"}';
GO

We then create a stored procedure to encapsulate the code used to call the API:

USE dbi_tools;
GO

CREATE OR ALTER PROCEDURE dbo.usp_BackupDatabase_WithZfsSnapshot
    @DatabaseName sysname,
    @BackupDirectory nvarchar(4000) = N'D:\Backups\'
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @Url nvarchar(4000) =
        N'https://192.168.1.110:8443/v1/sql-zfs/snapshot';

    DECLARE @Vmid int = 302;

    DECLARE @ZvolsJson nvarchar(max) =
        N'["sqlpool/pve/vm-302-disk-0"]';

    DECLARE @Stamp varchar(20) =
        REPLACE(REPLACE(CONVERT(varchar(19), SYSUTCDATETIME(), 126), '-', ''), ':', '') + 'Z';

    DECLARE @SafeDbName nvarchar(128) =
        REPLACE(REPLACE(REPLACE(@DatabaseName, N' ', N'_'), N'[', N''), N']', N'');

    DECLARE @SnapName nvarchar(128) =
        CONCAT(N'sql_', @SafeDbName, N'_', @Stamp);

    DECLARE @BackupFile nvarchar(4000) =
        CONCAT(@BackupDirectory, N'\', @SafeDbName, N'_', @Stamp, N'.bkm');

    DECLARE @Payload nvarchar(max) =
    (
        SELECT
            @DatabaseName AS [database],
            @Vmid AS [vmid],
            @SnapName AS [snapname],
            JSON_QUERY(@ZvolsJson) AS [zvols]
        FOR JSON PATH, WITHOUT_ARRAY_WRAPPER
    );

    DECLARE @ReturnCode int;
    DECLARE @Response nvarchar(max);
    DECLARE @SnapshotList nvarchar(max);

    SELECT @SnapshotList =
        STRING_AGG(CONCAT([value], N'@', @SnapName), N';')
    FROM OPENJSON(@ZvolsJson);

    DECLARE @MediaDescription nvarchar(max) =
        CONCAT(N'zfs|promox1|', @SnapshotList);

    DECLARE @Sql nvarchar(max);

    BEGIN TRY
        SET @Sql =
            N'ALTER DATABASE ' + QUOTENAME(@DatabaseName) +
            N' SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON;';

        EXEC sys.sp_executesql @Sql;

        EXEC @ReturnCode = sys.sp_invoke_external_rest_endpoint
            @url = @Url,
            @method = N'POST',
            @headers = N'{"Content-Type":"application/json","Accept":"application/json"}',
            @payload = @Payload,
            @credential = [https://192.168.1.110:8443/v1/sql-zfs/snapshot],
            @timeout = 30,
            @response = @Response OUTPUT;

        IF @ReturnCode <> 0
        BEGIN
            DECLARE @Err nvarchar(max) =
                CONCAT(N'ZFS snapshot API failed. ReturnCode=', @ReturnCode, N' Response=', @Response);
            THROW 51001, @Err, 1;
        END;

        SET @Sql =
            N'BACKUP DATABASE ' + QUOTENAME(@DatabaseName) + N'
              TO DISK = @BackupFile
              WITH METADATA_ONLY,
                   FORMAT,
                   MEDIANAME = @MediaName,
                   MEDIADESCRIPTION = @MediaDescription,
                   NAME = @BackupName;';

        EXEC sys.sp_executesql
            @Sql,
            N'@BackupFile nvarchar(4000),
              @MediaName nvarchar(128),
              @MediaDescription nvarchar(max),
              @BackupName nvarchar(128)',
            @BackupFile = @BackupFile,
            @MediaName = @SnapName,
            @MediaDescription = @MediaDescription,
            @BackupName = @SnapName;

        SELECT
            @DatabaseName AS database_name,
            @SnapName AS zfs_snapshot_name,
            @SnapshotList AS zfs_snapshots,
            @BackupFile AS metadata_backup_file,
            @MediaDescription AS media_description,
            @Response AS api_response;
    END TRY
    BEGIN CATCH
        IF DATABASEPROPERTYEX(@DatabaseName, 'IsDatabaseSuspendedForSnapshotBackup') = 1
        BEGIN
            SET @Sql =
                N'ALTER DATABASE ' + QUOTENAME(@DatabaseName) +
                N' SET SUSPEND_FOR_SNAPSHOT_BACKUP = OFF;';

            EXEC sys.sp_executesql @Sql;
        END;

        THROW;
    END CATCH
END;
GO

We then call the stored procedure:

EXEC dbi_tools.dbo.usp_BackupDatabase_WithZfsSnapshot
    @DatabaseName = N'StackOverflow',
    @BackupDirectory = N'D:\Backups\';

The backup was generated :

References

sp_invoke_external_rest_endpoint

Thank you. Amine Haloui

L’article SQL Server Snapshot Backup and Restore with Proxmox ZFS – REST API with SQL Server 2025 (3/3) est apparu en premier sur dbi Blog.

To address this issue, it is possible to automate the solution with PowerShell. The idea is to automate the different operations involved in the snapshot backup and restore process.

We will use two scripts:

• One script to perform the backups and create the snapshots.
• One script to perform the restores.

Backup process

Here is how the backup process works:

• We connect to the corresponding SQL Server instance.
• We change the state of the database using ALTER DATABASE … SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON. At this point, the I/Os are frozen.
• We connect to the hypervisor through SSH.
• We create the snapshot.
• We back up the database using BACKUP DATABASE … WITH METADATA_ONLY.
• We change the state of the database using ALTER DATABASE … SET SUSPEND_FOR_SNAPSHOT_BACKUP = OFF. At this point, the I/Os are unfrozen.

Powershell implementation (backup)

Here is the code used to perform the backup:

param(
    [string]$SqlInstance = "VM-WS25-SQL2",
    [string]$Database    = "StackOverflow",
    [string]$BackupDir   = "D:\Backups",
    [string]$PveHost     = "192.168.1.110",
    [string]$PveUser     = "MyUser",
    [string[]]$Zvols     = @("sqlpool/pve/vm-302-disk-0")
)

$Timestamp = Get-Date -Format "yyyyMMddTHHmmss"
$SnapName  = "sql_${Database}_${Timestamp}"

$DbSafe = $Database.Replace("]", "]]")
$BackupFile = Join-Path $BackupDir "${Database}_${Timestamp}.bkm"

$ZfsSnapshots = $Zvols | ForEach-Object { "$_@$SnapName" }
$ZfsSnapshotArgs = $ZfsSnapshots -join " "

$MediaDescription = "zfs|$PveHost|$ZfsSnapshotArgs"

$BackupFileSql = $BackupFile.Replace("'", "''")
$MediaSql = $MediaDescription.Replace("'", "''")

$connString = "Server=$SqlInstance;Database=master;Integrated Security=True;TrustServerCertificate=True;Application Name=ZFS-TSQL-Snapshot;"
$conn = New-Object System.Data.SqlClient.SqlConnection $connString

function Invoke-SqlNonQuery {
    param([string]$Sql)

    $cmd = $conn.CreateCommand()
    $cmd.CommandTimeout = 0
    $cmd.CommandText = $Sql
    [void]$cmd.ExecuteNonQuery()
}

try {
    $conn.Open()

    Write-Host "Freezing SQL database writes..."
    Invoke-SqlNonQuery "ALTER DATABASE [$DbSafe] SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON;"

    Write-Host "Taking ZFS snapshot on Proxmox..."
    ssh "$PveUser@$PveHost" "zfs snapshot $ZfsSnapshotArgs && zfs hold sqlsnap $ZfsSnapshotArgs"

    if ($LASTEXITCODE -ne 0) {
        throw "ZFS snapshot failed on $PveHost"
    }

    Write-Host "Writing SQL metadata backup..."

    Invoke-SqlNonQuery @"
BACKUP DATABASE [$DbSafe]
TO DISK = N'$BackupFileSql'
WITH METADATA_ONLY,
     MEDIADESCRIPTION = N'$MediaSql',
     NAME = N'$SnapName';
"@

    Write-Host "Snapshot backup completed:"
    Write-Host "  Snapshot: $ZfsSnapshotArgs"
    Write-Host "  Metadata: $BackupFile"
}
catch {
    Write-Warning $_

    try {
        Write-Warning "Attempting to unfreeze SQL database..."
        Invoke-SqlNonQuery "ALTER DATABASE [$DbSafe] SET SUSPEND_FOR_SNAPSHOT_BACKUP = OFF;"
    }
    catch {
        Write-Warning "Could not unfreeze cleanly. Check SQL Server error log."
    }

    throw
}
finally {
    $conn.Close()
}

Restore process

Here is how the restore process works:

• We connect to the corresponding SQL Server instance.
• We take the database offline.
• The volume dedicated to the StackOverflow database is taken offline.
• We connect to the hypervisor through SSH.
• We roll back the corresponding snapshot.
• We restore the database using the corresponding backup, which was created at the same time as the snapshot.

Powershell implementation (restore)

Here is the code used to perform the restore:

param(
    [string]$SqlInstance = "VM-WS25-SQL2",
    [string]$Database    = "StackOverflow",
    [string]$BackupFile  = "D:\Backups\StackOverflow_20260514T122642.bkm",
    [string]$SnapName    = "sql_StackOverflow_20260514T122642",
    [string]$PveHost     = "192.168.1.110",
    [string]$PveUser     = "MyUser",
    [string[]]$Zvols     = @("sqlpool/pve/vm-302-disk-0"),
    [string[]]$DatabaseDriveLetters = @("T"),
    [switch]$NoRecovery
)

$ErrorActionPreference = "Stop"

function Assert-SafeName {
    param(
        [string]$Value,
        [string]$Name,
        [string]$Pattern
    )

    if ($Value -notmatch $Pattern) {
        throw "$Name contained not allowed characters : $Value"
    }
}

function Normalize-DriveLetter {
    param([string]$DriveLetter)

    $letter = $DriveLetter.Trim().TrimEnd(":").ToUpperInvariant()

    if ($letter -notmatch '^[A-Z]$') {
        throw "Drive letter invalid : $DriveLetter"
    }

    return $letter
}

function Get-DiskForDriveLetter {
    param([string]$DriveLetter)

    $letter = Normalize-DriveLetter $DriveLetter

    $partition = Get-Partition -DriveLetter $letter -ErrorAction Stop
    $disk = $partition | Get-Disk -ErrorAction Stop

    return [pscustomobject]@{
        DriveLetter = $letter
        DiskNumber  = [int]$disk.Number
        IsOffline   = [bool]$disk.IsOffline
        FriendlyName = $disk.FriendlyName
        Size        = $disk.Size
    }
}

function Invoke-SshChecked {
    param([string]$Command)

    Write-Host "SSH $PveUser@$PveHost :: $Command"

    & ssh "$PveUser@$PveHost" "$Command"

    if ($LASTEXITCODE -ne 0) {
        throw "SSH command failed with code $LASTEXITCODE : $Command"
    }
}

function New-SqlConnection {
    $connString = "Server=$SqlInstance;Database=master;Integrated Security=True;TrustServerCertificate=True;Application Name=ZFS-TSQL-Restore-NoVmRestart;"
    return New-Object System.Data.SqlClient.SqlConnection $connString
}

function Invoke-SqlNonQuery {
    param([string]$Sql)

    $conn = New-SqlConnection

    try {
        $conn.Open()
        $cmd = $conn.CreateCommand()
        $cmd.CommandTimeout = 0
        $cmd.CommandText = $Sql
        [void]$cmd.ExecuteNonQuery()
    }
    finally {
        $conn.Close()
    }
}

function Invoke-SqlScalar {
    param([string]$Sql)

    $conn = New-SqlConnection

    try {
        $conn.Open()
        $cmd = $conn.CreateCommand()
        $cmd.CommandTimeout = 0
        $cmd.CommandText = $Sql
        return $cmd.ExecuteScalar()
    }
    finally {
        $conn.Close()
    }
}

function Set-DatabaseDisksOffline {
    param([object[]]$DiskInfos)

    $offlinedByScript = @()

    foreach ($diskInfo in ($DiskInfos | Sort-Object DiskNumber -Unique)) {
        if ($diskInfo.IsOffline) {
            Write-Host "Disque $($diskInfo.DiskNumber) déjà offline. Lecteur $($diskInfo.DriveLetter):"
            continue
        }

        Write-Host "Taking the Windows disk offline $($diskInfo.DiskNumber), drive $($diskInfo.DriveLetter):"
        Set-Disk -Number $diskInfo.DiskNumber -IsOffline $true

        $offlinedByScript += $diskInfo
    }

    return $offlinedByScript
}

function Set-DatabaseDisksOnline {
    param([object[]]$DiskInfos)

    foreach ($diskInfo in ($DiskInfos | Sort-Object DiskNumber -Unique)) {
        Write-Host "Bringing the Windows disk back online. $($diskInfo.DiskNumber), drive $($diskInfo.DriveLetter):"
        Set-Disk -Number $diskInfo.DiskNumber -IsOffline $false
    }

    Write-Host "Update-HostStorageCache..."
    Update-HostStorageCache
}

Assert-SafeName -Value $SnapName -Name "SnapName" -Pattern '^[A-Za-z0-9_.:-]{1,160}$'

foreach ($zvol in $Zvols) {
    Assert-SafeName -Value $zvol -Name "Zvol" -Pattern '^[A-Za-z0-9_.:/-]{1,240}$'
}

$DbQuoted = "[" + $Database.Replace("]", "]]") + "]"
$DbLiteral = $Database.Replace("'", "''")
$BackupFileSql = $BackupFile.Replace("'", "''")

$ZfsSnapshots = $Zvols | ForEach-Object { "$_@$SnapName" }
$ZfsSnapshotArgs = ($ZfsSnapshots | ForEach-Object { "'$_'" }) -join " "

$RecoveryOption = if ($NoRecovery) { "NORECOVERY" } else { "RECOVERY" }

$DatabaseDiskInfos = @()
$DisksOfflinedByScript = @()

Write-Host ""
Write-Host "Restore SQL Server from a ZFS snapshot, without restarting the VM"
Write-Host "SQL Instance : $SqlInstance"
Write-Host "Database     : $Database"
Write-Host "BackupFile   : $BackupFile"
Write-Host "DB volumes   : $($DatabaseDriveLetters -join ', ')"
Write-Host "Snapshots    :"
$ZfsSnapshots | ForEach-Object { Write-Host "  $_" }
Write-Host ""

try {
    Write-Host "Checking ZFS snapshots..."
    Invoke-SshChecked "zfs list -H -t snapshot -o name $ZfsSnapshotArgs >/dev/null"

    Write-Host "Identifying Windows disks containing SQL Server files..."
    foreach ($driveLetter in $DatabaseDriveLetters) {
        $diskInfo = Get-DiskForDriveLetter $driveLetter
        $DatabaseDiskInfos += $diskInfo

        Write-Host "Drive $($diskInfo.DriveLetter): -> Windows disk $($diskInfo.DiskNumber) [$($diskInfo.FriendlyName)]"
    }

    $backupDrive = $null
    if ($BackupFile -match '^([A-Za-z]):\\') {
        $backupDrive = Normalize-DriveLetter $Matches[1]

        try {
            $backupDiskInfo = Get-DiskForDriveLetter $backupDrive
            $targetDiskNumbers = @($DatabaseDiskInfos | ForEach-Object { $_.DiskNumber } | Select-Object -Unique)

            if ($targetDiskNumbers -contains $backupDiskInfo.DiskNumber) {
                throw @"
The backup file $BackupFile is located on drive $backupDrive, which is on the same Windows disk as the SQL Server data volume.
Taking the data disk offline would make the .bkm file inaccessible, and a rollback could also make the .bkm file disappear.
Move the .bkm file to C:, a network share, or another disk that is not rolled back.
"@
            }
        }
        catch {
            throw
        }
    }

    Write-Host "Checking whether the SQL Server database exists..."
    $DbExists = Invoke-SqlScalar "SELECT CASE WHEN DB_ID(N'$DbLiteral') IS NULL THEN 0 ELSE 1 END;"

    if ($DbExists -eq 1) {
        Write-Host "Taking database $Database OFFLINE..."
        Invoke-SqlNonQuery @"
ALTER DATABASE $DbQuoted SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
ALTER DATABASE $DbQuoted SET OFFLINE WITH ROLLBACK IMMEDIATE;
"@
    }
    else {
        Write-Host "Database $Database does not exist in SQL Server. Continuing with disk offline and ZFS rollback."
    }

    Write-Host "Taking Windows disks containing MDF/LDF files offline..."
    $DisksOfflinedByScript = Set-DatabaseDisksOffline -DiskInfos $DatabaseDiskInfos

    Write-Host "Rolling back ZFS snapshot..."
    $RollbackCommands = ($ZfsSnapshots | ForEach-Object { "zfs rollback -r '$_'" }) -join "; "
    Invoke-SshChecked "set -e; $RollbackCommands"

    Write-Host "Bringing Windows disks back online..."
    Set-DatabaseDisksOnline -DiskInfos $DisksOfflinedByScript
    $DisksOfflinedByScript = @()

    Write-Host "Short pause to let Windows and SQL Server detect the restored disk state..."
    Start-Sleep -Seconds 5

    Write-Host "Restoring SQL Server metadata-only backup..."

    $RestoreSql = @"
RESTORE DATABASE $DbQuoted
FROM DISK = N'$BackupFileSql'
WITH METADATA_ONLY,
     REPLACE,
     $RecoveryOption;
"@

    Invoke-SqlNonQuery $RestoreSql

    if (-not $NoRecovery) {
        Write-Host "Setting database back to MULTI_USER..."
        Invoke-SqlNonQuery @"
ALTER DATABASE $DbQuoted SET MULTI_USER;
"@
    }

    Write-Host ""
    Write-Host "Restore completed."
    Write-Host "Database : $Database"
    Write-Host "Snapshot : $SnapName"
    Write-Host "Backup   : $BackupFile"
}
catch {
    Write-Warning "Restore failed: $_"

    if ($DisksOfflinedByScript.Count -gt 0) {
        try {
            Write-Warning "Attempting to bring disks offlined by the script back online..."
            Set-DatabaseDisksOnline -DiskInfos $DisksOfflinedByScript
            $DisksOfflinedByScript = @()
        }
        catch {
            Write-Warning "Unable to automatically bring the disks back online. Check with Get-Disk."
        }
    }

    try {
        $DbExistsAfterError = Invoke-SqlScalar "SELECT CASE WHEN DB_ID(N'$DbLiteral') IS NULL THEN 0 ELSE 1 END;"

        if ($DbExistsAfterError -eq 1 -and -not $NoRecovery) {
            Write-Warning "Attempting to set the database back ONLINE/MULTI_USER..."
            Invoke-SqlNonQuery @"
ALTER DATABASE $DbQuoted SET ONLINE;
ALTER DATABASE $DbQuoted SET MULTI_USER;
"@
        }
    }
    catch {
        Write-Warning "Unable to automatically set the database back ONLINE/MULTI_USER."
    }

    throw
}

What does it look like?

We start the backup process:

We verify that the snapshot is present:

We verify that the backup is present:

We drop the StackOverflow database:

We start the restore process:

The database is available again. The restore took only a few seconds for a database of approximately 200 GB.

Major drawbacks

In my case, the solution is executed from the SQL Server itself. Ideally, it should rather be hosted on another server or client machine. We could also imagine running these scripts from a scheduler such as RedDeck, for example.

During the database restore, the database is switched to SINGLE_USER mode. This could be an issue if the applications using the database reconnect very frequently. A better approach would probably be to explicitly terminate the active sessions using the KILL command.

We have also not yet covered the use of a REST API.

Thank you. Amine Haloui

L’article SQL Server Snapshot Backup and Restore with Proxmox ZFS – Powershell implementation (2/3) est apparu en premier sur dbi Blog.

Error :  We couldn’t connect to the Analysis Services server. Make sure you’ve entered the connection string correctly.

However, this issue did not occur in Power BI Desktop.

In Power BI, several data loading modes are available. Import mode loads data into the Power BI model, which usually provides faster performance and richer modeling capabilities. DirectQuery mode does not store the data in the model instead, each interaction sends queries to the source system in real time. Import is generally better for speed and flexibility, while DirectQuery is useful when data must stay in the source or remain near real-time. The trade-off is that DirectQuery depends more heavily on source performance, network latency, and source-system limitations.

Configuration

At first glance, one might think that the corresponding report is trying to connect to an SSAS service and that there is a connectivity issue between Power BI Report Server and a SQL Server Analysis Services instance.

However, after reviewing the data source, there was no connection to SSAS:

We did not have this type of configuration:

The questions that arise

Why are we getting an error message even though the report is not trying to connect to a SQL Server Analysis Services instance?

Why is our client seeing this error message and unable to query the report?

Troubleshooting

By reviewing the Power BI Report Server logs, it was possible to see this type of message:

Failed to get CSDL. —> MsolapWrapper.MsolapWrapperException: Failure encountered while getting schema.

CannotRetrieveModelException: An error occurred while loading the model… Verify that the connection information is correct and that you have permissions to access the data source.

It is also possible to retrieve some information from the ExecutionLog3 table:

Indeed,  whenever a Power BI report is rendered or a scheduled refresh is executed, new entries are written to the ExecutionLog3 table. These entries can be queried through the ExecutionLog3 view in the Report Server catalog database. The ConceptualSchema event corresponds to a user viewing the report.

When querying the Event Viewer, it returned these errors at the time we tried to query the report:

More details about the first errors

We have two error messages that seem to point in two different directions. In reality, the first error messages are not very useful and appear because although the error message refers to Analysis Services, the report was not connecting to an external SSAS instance. Power BI Report Server uses an internal Analysis Services engine to load and query Power BI report models. Therefore, the error was raised by the internal PBIRS Analysis Services engine, not by a standalone SQL Server Analysis Services instance.

Power BI Report Server may report an Analysis Services-related error even when the report does not connect to an external SSAS instance. This is because PBIRS uses an internal Analysis Services engine to host and execute the Power BI semantic model behind the report. In DirectQuery mode, the data remains in SQL Server, but the report model, metadata, relationships, measures, and DAX queries are still processed through this internal engine.

When a user opens the report, PBIRS asks this local Analysis Services process to load the model and generate the queries sent to SQL Server.

Therefore, if the internal engine fails while loading the model, validating metadata, or connecting to the SQL Server data source, the error may mention Analysis Services. This does not mean that the report is connected to a standalone SSAS instance.

More details about the second errors

This was the second error that pointed us in the right direction to actually resolve the issue. After looking at it more closely, we started considering connection encryption and certificates. This problem is documented, and several solutions are available.

Indeed, the SQL Server instance queried to retrieve the data did not have a certificate issued by a trusted certificate authority. It was using a self-generated certificate.

This can lead to errors such as the ones mentioned above, or errors like the following:

Microsoft SQL: A connection was successfully established with the server, but then an error occurred during the login process. Provider: SSL Provider, error: 0 – The certificate chain was issued by an authority that is not trusted.

Solutions

We had at least three options to resolve this issue:

• Change the connection mode to Import
• Install a certificate issued by a trusted certificate authority however this would represent a major change
• Create a new environment variable on the Power BI Report Server

The client chose the easiest solution to implement: creating the corresponding environment variable.

We then restarted the corresponding Power BI Report Server service and this resolved the issue.

References :

https://learn.microsoft.com/en-us/power-bi/report-server/scheduled-refresh-troubleshoot

https://learn.microsoft.com/en-us/power-query/connectors/sql-server#sql-server-certificate-isnt-trusted-on-the-client-power-bi-desktop-or-on-premises-data-gateway

Thank you. Amine Haloui

L’article A Misleading SSAS Error in Power BI Report Server When Using DirectQuery Mode est apparu en premier sur dbi Blog.

The configuration parameter “io_workers” is gone and four additional parameters show up to control this:

postgres=# \dconfig io_*work*
 List of configuration parameters
         Parameter         | Value 
---------------------------+-------
 io_max_workers            | 8
 io_min_workers            | 2
 io_worker_idle_timeout    | 1min
 io_worker_launch_interval | 100ms
(4 rows)

“io_min_workes” (as the name implies) controls how many workers are available by default, which is two:

postgres@:/home/postgres/ [DEV] ps -ef | grep postgres | grep worker | grep -v grep
postgres    8564    8562  0 06:34 ?        00:00:00 postgres: pgdev: io worker 0
postgres    8565    8562  0 06:34 ?        00:00:00 postgres: pgdev: io worker 1

“io_max_workers” (again, as the name implies) controls the maximum worker processes which can be launched for the whole instance.

To see that dynamic startup of workers in action lets create a simple table containing twenty million rows:

postgres=# create table t ( a int, b text, c timestamptz );
CREATE TABLE
postgres=# insert into t select i, i::text, now() from generate_series(1,20000000) i;
INSERT 0 2000000

While watching the workers in a separate session:

postgres@:/home/postgres/ [DEV] watch "ps -ef | grep postgres | grep worker | grep -v grep"

Every 2.0s: ps -ef | grep postgres | grep worker | grep -v grep               pgbox.it.dbi-services.com: 06:52:20 AM
                                                                                                       in 0.022s (0)
postgres    8564    8562  0 06:34 ?        00:00:00 postgres: pgdev: io worker 0
postgres    8565    8562  0 06:34 ?        00:00:00 postgres: pgdev: io worker 1

… and doing a count(*) over the whole table in session one:

postgres=# select count(*) from t;
  count   
----------
 20000000
(1 row)

… you’ll notice that an additional worker (io worker 2) shows up in the second session watching the processes (maybe you have to play a bit with the number of rows depending on your configuration of PostgreSQL):

Every 2.0s: ps -ef | grep postgres | grep worker | grep -v grep               pgbox.it.dbi-services.com: 07:02:40 AM
                                                                                                       in 0.018s (0)
postgres    8564    8562  0 06:34 ?        00:00:02 postgres: pgdev: io worker 0
postgres    8565    8562  0 06:34 ?        00:00:00 postgres: pgdev: io worker 1
postgres   11914    8562  0 07:02 ?        00:00:00 postgres: pgdev: io worker 2

Once this additional worker is idle for one minute it will disappear and we’re back to two worker processes:

Every 2.0s: ps -ef | grep postgres | grep worker | grep -v grep               pgbox.it.dbi-services.com: 07:04:24 AM
                                                                                                       in 0.020s (0)
postgres    8564    8562  0 06:34 ?        00:00:02 postgres: pgdev: io worker 0
postgres    8565    8562  0 06:34 ?        00:00:00 postgres: pgdev: io worker 1

This is controlled by “io_worker_idle_timeout” and the default is one minute.

The remaining configuration knob is “io_worker_launch_interval”, and this is the interval at which additional workers can be launched. The reason behind this is, that not too many workers will be launched at once.

This will make tuning the workers easier, compared to PostgreSQL 18. Again, thanks to all involved, the commit is here.

L’article PostgreSQL 19: Dynamically adjust the I/O worker pool est apparu en premier sur dbi Blog.

This is maybe not a feature a lot of people have waited for but it makes two tasks a lot easier:

• As mentioned above: pg_verifybackup can now read from WAL in tar and compressed files and therefore can do WAL verification
• When you have WAL in a tar or compressed file and you know what you’re looking for you do not need to manually extract those archives before using pg_waldump

To see that in action once can create a tar or compressed backup with pb_basebackup:

postgres@:/home/postgres/ [pgdev] mkdir /var/tmp/dummy
postgres@:/home/postgres/ [pgdev] pg_basebackup --checkpoint=fast --format=t --pgdata=/var/tmp/dummy
postgres@:/home/postgres/ [pgdev] ls -la /var/tmp/dummy
total 128476
drwxr-xr-x. 1 postgres postgres        66 May 11 06:36 .
drwxrwxrwt. 1 root     root           762 May 11 06:33 ..
-rw-------. 1 postgres postgres    149515 May 11 06:36 backup_manifest
-rw-------. 1 postgres postgres 114619904 May 11 06:36 base.tar
-rw-------. 1 postgres postgres  16778752 May 11 06:36 pg_wal.tar

Looking at the PostgreSQL log file while the backup is running gives us a LSN we can give to pg_waldump:

2026-05-11 06:36:18.397 CEST - 2 - 1731 -  - @ - 0LOG:  checkpoint complete: fast force wait: wrote 2 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 1 removed, 0 recycled; write=0.002 s, sync=0.005 s, total=0.019 s; sync files=4, longest=0.003 s, average=0.002 s; distance=16384 kB, estimate=16384 kB; lsn=0/0D000088, redo lsn=0/0D000028

postgres@:/home/postgres/ [pgdev] pg_waldump --path=/var/tmp/dummy/pg_wal.tar -s "0/0D000088" 
rmgr: XLOG        len (rec/tot):    122/   122, tx:          0, lsn: 0/0D000088, prev 0/0D000050, desc: CHECKPOINT_ONLINE redo 0/0D000028; tli 1; prev tli 1; fpw true; wal_level replica; logical decoding false; xid 0:729; oid 16420; multi 1; offset 1; oldest xid 684 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 729; checksums on; online
rmgr: Standby     len (rec/tot):     54/    54, tx:          0, lsn: 0/0D000108, prev 0/0D000088, desc: RUNNING_XACTS nextXid 729 latestCompletedXid 728 oldestRunningXid 729; dbid: 0
rmgr: XLOG        len (rec/tot):     34/    34, tx:          0, lsn: 0/0D000140, prev 0/0D000108, desc: BACKUP_END 0/0D000028
rmgr: XLOG        len (rec/tot):     24/    24, tx:          0, lsn: 0/0D000168, prev 0/0D000140, desc: SWITCH 
pg_waldump: error: could not find WAL "00000001000000000000000E" in archive "pg_wal.tar

This helps pg_verifybackup fully verify a backup (in previous versions you had to use “–no-parse-wal”):

postgres@:/home/postgres/ [pgdev] pg_verifybackup --progress /var/tmp/dummy/
111933/111933 kB (100%) verified
backup successfully verified

As usual, thanks to all involved.

L’article PostgreSQL 19: pg_waldump can now read from archives est apparu en premier sur dbi Blog.

For looking at this we need a simple setup, so we initialize two new PostgreSQL 19 clusters and connect them with postgres_fdw:

postgres@:/home/postgres/ [pgdev] initdb --version
initdb (PostgreSQL) 19devel
postgres@:/home/postgres/ [pgdev] initdb --pgdata=/var/tmp/pg1
postgres@:/home/postgres/ [pgdev] initdb --pgdata=/var/tmp/pg2
postgres@:/home/postgres/ [pgdev] echo "port=8888" >> /var/tmp/pg1/postgresql.auto.conf 
postgres@:/home/postgres/ [pgdev] echo "port=8889" >> /var/tmp/pg2/postgresql.auto.conf 
postgres@:/home/postgres/ [pgdev] pg_ctl --pgdata=/var/tmp/pg1/ start
postgres@:/home/postgres/ [pgdev] pg_ctl --pgdata=/var/tmp/pg2/ start
postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "create extension postgres_fdw"
CREATE EXTENSION
postgres@:/home/postgres/ [pgdev] psql -p 8889 -c "create table t ( a int, b text, c timestamptz )"
CREATE TABLE
postgres@:/home/postgres/ [pgdev] psql -p 8889 -c "insert into t select i, md5(i::text), now() from generate_series(1,1000000) i"
INSERT 0 1000000
postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "create server srv_pg2 foreign data wrapper postgres_fdw options(port '8889', dbname 'postgres')"
CREATE SERVER
postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "create user mapping for postgres server srv_pg2 options (user 'postgres', password 'postgres')"
CREATE USER MAPPING
postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "create foreign table ft (a int, b text, c timestamptz) server srv_pg2 options (schema_name 'public', table_name 't')"
CREATE FOREIGN TABLE
postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "select count(*) from ft"
  count  
---------
 1000000
(1 row)

What we have now is one table in the cluster on port 8889 and this table is attached as a foreign table in the cluster on port 8888.

We already have statistics on the source table in the cluster on port 8889:

postgres@:/home/postgres/ [pgdev] psql -p 8889 -c "select reltuples::bigint from pg_class  where relname = 't'"

 reltuples 
-----------
   1000000
(1 row)

… but we do not have any statistics on the foreign table in the cluster on port 8888:

postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "select reltuples::bigint from pg_class  where relname = 'ft'"

 reltuples 
-----------
        -1

(1 row)

Only after manually analyzing the foreign table the statistics show up:

postgres@:/home/postgres/ [DEV] psql -p 8888 -c "analyze ft"
ANALYZE
postgres@:/home/postgres/ [DEV] psql -p 8888 -c "select reltuples::bigint from pg_class  where relname = 'ft'"

 reltuples 
-----------
   1000000
(1 row)

The issue that can arise with these local statistics is, that they probably become outdated when the source table is modified:

postgres@:/home/postgres/ [pgdev] psql -p 8889 -c "insert into t select i, md5(i::text), now() from generate_series(1000001,2000000) i"
INSERT 0 1000000
postgres@:/home/postgres/ [DEV] psql -p 8889 -c "select reltuples::bigint from pg_class  where relname = 't'"

 reltuples 
-----------
   2000000
(1 row)

postgres@:/home/postgres/ [DEV] psql -p 8888 -c "select reltuples::bigint from pg_class  where relname = 'ft'"

 reltuples 
-----------
   1000000
(1 row)

As you can see, the row counts do not match anymore. Once the local statistics are gathered we again have the same picture on both sides:

postgres@:/home/postgres/ [DEV] psql -p 8888 -c "analyze ft"
ANALYZE
postgres@:/home/postgres/ [DEV] psql -p 8888 -c "select reltuples::bigint from pg_class  where relname = 'ft'"

 reltuples 
-----------
   2000000
(1 row)

One way to avoid this issue even before PostgreSQL 19 is to tell postgres_fdw to run analyze on the remote table and to use those statistics:

postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "alter foreign table ft options ( use_remote_estimate 'true' )"

In this case the local statistics will not be used but of course this comes with the overhead of the additional analyze on the remote side.

From PostgreSQL 19 there is another option:

postgres@:/home/postgres/ [pgdev] psql -p 8888 -c "alter foreign table ft options ( restore_stats 'true' )"
ALTER FOREIGN TABLE

This option tells postgres_fdw to import the statistics from the remote side and store them locally. If that fails it will run analyze as above, the commit message nicely explains this:

Add support for importing statistics from remote servers.

Add a new FDW callback routine that allows importing remote statistics
for a foreign table directly to the local server, instead of collecting
statistics locally.  The new callback routine is called at the beginning
of the ANALYZE operation on the table, and if the FDW failed to import
the statistics, the existing callback routine is called on the table to
collect statistics locally.

Also implement this for postgres_fdw.  It is enabled by "restore_stats"
option both at the server and table level.  Currently, it is the user's
responsibility to ensure remote statistics to import are up-to-date, so
the default is false.

As usual, thanks to all involved.

L’article PostgreSQL 19: Importing statistics from remote servers est apparu en premier sur dbi Blog.