I started the second day of PGConf.EU with a session about vacuum and autovacuum, presented by Teresa Lopes.
After a short introduction to how MVCC works, she presented the different elements related to autovacuum:
What are the different phases of autovacuum? What can trigger autovacuum? Which parameters can influence autovacuum? All these points were explained in a simple and clear way during the presentation.
Teresa also presented several reasons why autovacuum might not start, including a long-running transaction, an abandoned replication slot or an abandoned prepared transaction.

The second half of the presentation focused on new autovacuum features in PostgreSQL 19, for example:

• The scoring system for autovacuum, based on criteria such as XID/MXID age and the number of inserted, updated or deleted tuples.
• The possibility to use parallel workers for index vacuuming and cleanup.
• Improved ways to observe and monitor autovacuum, especially through new views and new logging parameters.

The session ended with a look at how to recover disk space in tables after autovacuum has cleaned them up, especially with REPACK, which also has a CONCURRENTLY option that allows other transactions to use the table while it is being repacked.

Zero downtime?

Speaking about REPACK, this new feature was also presented by Rajni Baliyan during her presentation about how to achieve zero downtime with PostgreSQL 19.
REPACK CONCURRENTLY allows tables to be reorganized and disk space to be recovered without blocking normal database operations for the entire process. Unlike VACUUM FULL, which blocks access to the table during the operation, REPACK CONCURRENTLY allows applications to continue reading and writing data while a new copy of the table and its indexes is being created.
In the background, PostgreSQL uses logical decoding to capture changes made to the original table during this process and applies them to the new copy. At the end, it briefly takes an exclusive lock to replace the old table and indexes with the new ones.

An other interesting feature is the new logical sequence synchronization worker, which helps keep sequence values synchronized between the publisher and the subscriber during logical replication.
Before PostgreSQL 19, sequence values were not automatically synchronized through logical replication. This could cause problems during major version upgrades, as sequences on the new server could have outdated values and generate duplicate IDs.
With PostgreSQL 19, sequences can now be synchronized automatically when a subscription is created. The REFRESH SEQUENCES command can also be used to synchronize them again when needed.

As you may have guess, the goal of this presentation was to show how new features can help reduce service interruptions during database maintenance and remove the need for maintenance windows. The presentation focused on making database operations easier and safer while keeping services available for users.
It was an interesting topic, especially for production environments where downtime must be avoided as much as possible.

Modern Backups for PostgreSQL

I then attended a session about pg_hardstorage, presented by Hans-Jürgen Schönig.
pg_hardstorage is a modern backup solution for PostgreSQL, designed especially for large environments and critical systems.
Unlike traditional backup tools, it uses the PostgreSQL replication protocol to retrieve a base backup and continuously stream WAL files. It does not require an agent or direct shell access on the database server. The tool runs on a separate backup host.

One interesting feature is its storage system. Each backup is treated as a full backup, but duplicate data blocks are stored only once. This reduces storage requirements without the complexity of traditional incremental backup chains.
The continuous WAL streaming also allows point-in-time recovery, making it possible to restore a database to a specific moment in time.

Another advantage is that pg_hardstorage can store backups on a filesystem or S3-compatible object storage. It also supports different PostgreSQL environments, including bare-metal servers, Patroni clusters and Kubernetes deployments.

Overall, it was an interesting presentation about an alternative approach to PostgreSQL backups, with a strong focus on simplicity, storage efficiency and reliable recovery.

Patroni the right way

I then attended a session presented by Alexander Kukushkin and Polina Bungina, two of the main contributors to Patroni.
The presentation explained how Patroni really works behind the scenes, with a special focus on automatic failover and the challenges of managing a high-availability PostgreSQL cluster.

One important topic was the risk of split-brain, when two PostgreSQL nodes believe they are the primary at the same time. The speakers explained why a simple two-node setup is not enough and why quorum and fencing mechanisms are important to prevent this situation.
They also explained how Patroni uses a Distributed Configuration Store (DCS), such as etcd, to coordinate the cluster, manage the leader election and keep track of the state of each node.
The presentation also covered some important features, such as synchronous replication, standby clusters and DCS failsafe mode. Finally, they presented some upcoming features, including site-aware failover, which will help Patroni make better decisions when a cluster is spread across several data centers.

This was a very nice session to better understand the internal mechanisms of Patroni and the challenges of building a reliable PostgreSQL high-availability architecture.

Data Quality

Due to meetings with my customers, I unfortunately couldn’t attend any other sessions on Wednesday. However, I just managed to catch my colleague Hervé Schweitzer presentation of our OMrun tool during the “Silver Sponsor Lightning Talks”.

Are you interested in data quality?
Take a look here to learn more about OMrun: https://www.dbi-services.com/fr/produits/omrun/

New communication Hub for the PostgreSQL Community

On Thursday, I attended a session presented by Jack Bonatakis about Hackorum, a tool designed to make PostgreSQL mailing lists easier to use for both experts and newcomers.
PostgreSQL development relies heavily on mailing lists, but following long email discussions can be difficult, especially when there are many messages.
Hackorum provides a modern, forum-style web interface on top of the existing mailing lists. It does not replace them. Instead, it makes discussions easier to read, follow and organize. Some of its main features are:

• Better discussion view: messages are grouped into topics, making long conversations easier to follow.
• Read tracking: you can see which messages you have already read and continue where you stopped.
• Commitfest integration: information about patches and their status is available directly alongside the related discussions.
• Tags, notes and favourites: users can organize discussions, save important topics and add personal notes.
• Team collaboration: reading status, tags and notes can be shared with other team members.

Compared to traditional mailing lists, Hackorum offers a more visual and user-friendly interface. It helps reduce the number of emails to manage and makes it easier to follow several discussions at the same time. The integration with Commitfest is also useful for PostgreSQL developers who want to follow patches and their progress.
One important point is that the mailing lists remain the main source of information. Hackorum simply provides a more convenient way to access and organize their content.

I found this an interesting initiative that could make the PostgreSQL community more accessible, especially for newcomers who may find the traditional mailing lists difficult to follow.

Monitoring

I finished the conference with a presentation by Jimmy Angelakos about radar and radar-analyst, two tools designed to simplify PostgreSQL diagnostics.
The idea behind radar is simple: instead of installing monitoring agents and collecting information over a long period, you can run a single command to collect a snapshot of the PostgreSQL server and its operating system.
The tool is agentless and does not require any additional dependencies. It creates a ZIP archive containing useful diagnostic information, including PostgreSQL configuration, database statistics, replication status, table and index information, as well as system resources such as CPU, memory and disk I/O.

The second tool, radar-analyst, analyzes this archive and helps identify potential problems. It uses predefined rules to check different areas, including PostgreSQL configuration, workload, replication and operating system settings. Each finding includes an explanation and recommendations to help fix the problem.

Compared to traditional monitoring solutions, this approach has several advantages. There is no need to install or maintain an agent, and a diagnostic snapshot can be collected when a problem occurs, even if no monitoring was configured beforehand.

I found this to be a practical approach to PostgreSQL troubleshooting. It can be particularly useful for support teams and DBAs who need to quickly understand the state of a database server without spending hours collecting information manually.

L’article PGConf.EU 2026 – Day 2 & 3 est apparu en premier sur dbi Blog.

Something surprised me right at the start. During the welcome session on the first day, the organizers asked who was attending for the first time, and around 30% of the room raised their hand! I don’t know exactly what it means, maybe more companies are moving to PostgreSQL, maybe more young people are joining the community. But for sure it means a lot of new people, new ideas and new conversations, and that’s a very good sign for PostgreSQL.

Open Source projects you can use directly

Many developers presented their own open source PostgreSQL projects. After the session, you can directly download them for free, use them, and contribute to them!

For passionate people, this is a real added value. You can play with the tool right away, without having to contact a salesperson who will not leave you alone afterwards . And if it fits your needs, you can simply use it in your daily work, without asking for a budget first!

I tried to count them, and I found more than 15 sessions where the project is directly available on GitHub. Some examples:

• pg-storage-visualizer (storage internals)
• pgstream (CDC)
• pg_track_optimizer (bad query plans)
• radar (diagnostic snapshots)
• pg_acm (access control)

And many more.

Real experience from the field

Many technical sessions were about real projects, where the speakers shared their experience with PostgreSQL.

I noticed that there are still a lot of sessions about migrating from Oracle to PostgreSQL. I thought the rush was over, but no, there are still many use cases. What changed is the type of database: today more and more big and critical Oracle environments are starting to leave Oracle. But in most cases these are still databases without too much code inside (PL/SQL). This remains the biggest challenge to leave Oracle.

Not only Oracle! we also see more and more customers migrating from DB2 to PostgreSQL, often companies that never made the jump to Oracle in the past, and now go directly to PostgreSQL

Current trends from the community

There were a lot of sessions about the new PostgreSQL 19 features. Sometimes the presentations were a little bit redundant (REPACK, REPACK, and again REPACK ). But at least we know now which feature will push people to migrate to PostgreSQL 19: REPACK in core!

Sponsor booths

Between the sessions, the sponsor area was also a very good place to meet people, where I had the opportunity to visit some of our partners, like dbtune, and discuss new topics and opportunities with them. These discussions are often as valuable as the sessions themselves, because you get direct feedback and you can talk about real customer use cases!

And AI?

Today, everybody talks about AI. At the conference, it was mainly a topic for the sponsors and not so much for the community (with some exceptions, like the fun “Dungeon Master with Postgres and MCP” session). More and more services come with AI integration, but I don’t think many of them are already used in production. And above all, I didn’t hear any real customer feedback about them yet!

Evening in Valencia

PGconf.eu it’s also a good opportunity to discover a new place in Europe. I have to say, that I was very positively surprised by Valencia! it’s a very pleasant city, with a lot of culture and history. And after the sessions, it’s also the perfect place to go out for dinner with other participants, where the discussions around PostgreSQL will continue with a beer

Conclusion

Overall, I would say that PGConf.EU is still the best place to see where PostgreSQL is going, and above all to meet the people who build it. For dbi services it was also a great event as a Silver Sponsor, and I also had the chance to present OMrun in the sponsor lightning talks.

If you have never been there, I can only recommend it. See you next year!

We are hiring PostgreSQL consultants !

If you want to work with PostgreSQL every day and go to conferences like PGConf.EU, join us! At dbi services we are looking for PostgreSQL consultants to work with our customers on HA setups, performance tuning, migrations, Kubernetes and much more.

Open Infrastructure Consultant

L’article Why attend PGConf.EU? My feedback from Valencia est apparu en premier sur dbi Blog.

This year, the European PostgreSQL conference, PGConf.EU, takes place at the Palacio de Congresos in Valencia, Spain. I am lucky to be able to attend it with some colleagues from dbi services. Let’s see what’s going on there…

I started the first day with a quick tour of the both of the different conference sponsors.

I then joined the main auditorium for the welcome session, presented by Magnus Hagander.
After sharing some information about the organization and how the conference would run, Magnus gave a nice overview of the history of PostgreSQL and PGConf. I found it very interesting to discover the history and evolution of the different conferences.
Even Slonik came on stage to give us a shy hello

During the coffee break, I went to watch a live demo at the DBtune stand.
DBtune is a tool that uses AI to automatically optimize PostgreSQL performance. It analyzes the database workload and adjusts server parameters to improve performance and resource usage, with the goal of reducing costs and manual work.

Extensions

The morning continued with a session presented by Gülcin Yildirim Jelinak and Petr Jelinek. The goal of their presentation was to explain the main strengths of PostgreSQL’s execution model for OLTP workloads, such as MVCC, vacuuming and WAL management. They also explained the limitations of these mechanisms when dealing with large amounts of data, either for writing or for analytics.
The two speakers explained that, in some configurations, PostgreSQL can work as a transactional control layer and an SQL interface, while specialized systems handle execution tasks that are different from traditional OLTP workloads. Many of these solutions are available as extensions, such as pg_search, pg_lake, pg_duckdb and pg_clickhouse. These extensions integrate external execution engines directly with PostgreSQL.

Speaking of extensions, the next session I attended was entirely dedicated to this topic. It was presented by Ebru Aydin Gol, who talked about different approaches and best practices for managing changes and updates to extensions while keeping backward compatibility as much as possible. The general idea is to systematically classify all changes according to their impact on backward compatibility and on the behavior experienced by users.

New features and pgBackRest

I then went to see my former colleague Franck Pachot, who presented some new features of PostgreSQL 19, which we hope will be released very soon!
The new features were presented through live demos. Many of them focused on performance improvements, especially thanks to changes in the query planner.
Franck finished his session by talking about pg_durable, an extension that allows workflows to be defined and executed directly in the database. If a crash happens, it can resume the workflow from where it stopped. This is an interesting extension that could make tools such as pg_cron or other external orchestrators unnecessary for some use cases.

After the afternoon break, I attended a session that was a little less technical but just as interesting. It was about the different ways to implement pgBackRest in high-availability environments.
Stefan Ferot presented several pgBackRest deployment patterns, starting with the simplest solution (one PostgreSQL host connected to one pgBackRest repository host) and moving to the most robust solution.
The most robust architecture uses cloud storage, such as S3, to store archived WAL files directly, together with a repository host that pulls backups from the standby server. This architecture offers several advantages. It removes the SPOF related to on-premises WAL archiving and provides centralized backup orchestration through the repository host.

I particularly appreciated that Stefan highlighted the different pgBackRest parameters to consider on both the PostgreSQL hosts and the repository host for each architecture he presented. Despite this, I would have liked to see a demo showing the resilience of the most robust architecture.

Migration

My day ended with a presentation about an Oracle to PostgreSQL migration project of a platform providing more than 500 services to citizens of an Indian state. Hari Kira explained how they handled the rewriting and migration of Oracle-dependent code, the use of ora2pg and its limitations, as well as the main strategic decisions and different project steps.
In the end, everything was migrated to a single PostgreSQL instance, with Patroni, etcd and HAProxy providing high availability.

A few thoughts after day one

My first impressions of PGConf.EU are very positive. The atmosphere is relaxed and friendly, and the conference is very well organized. The technical sessions are very interesting, with several talks about the new features coming in PostgreSQL 19. I also noticed that there were very few sessions about AI, which is quite surprising nowadays.
Overall, this conference is a great opportunity to learn new things and share experiences with the PostgreSQL community.

Let’s go for the second day!

L’article PGConf.EU 2026 – Day 1 est apparu en premier sur dbi Blog.

When discussing agentic AI, we spend a lot of time on the model. Which one should we use? How much context does it support? Can it choose the right tool? These are useful questions. But from a my perspective, there are a few others I would ask quite early. Where does the agent store its state? Who can access the information it retrieves? What happens when two agents update the same record? Can we recover after a bad change? And can we explain which data was used to produce an answer? What governance do we need to apply ?
In an upcoming essay, Deciding in the Storm, I am exploring how organizations can make useful decisions while AI keeps changing. Here, I focus on the application underneath: where its knowledge and workflow state live, who may change them, and what happens when several users or agents depend on them at once. These are questions database teams already know how to investigate.

That is good news. We already have useful tools and operational experience, but we need to apply them properly.

What a knowledge format gives us

On 12 June 2026, Google Cloud introduced the Open Knowledge Format, or OKF. The initial specification represents knowledge as Markdown files with YAML frontmatter and conventions for linking and organizing them. Table descriptions, metric definitions, runbooks and business rules can be exchanged in a format that both people and agents can read. You can review the files, version them in Git and move them between tools.
I like that approach. We do not need another proprietary API just to explain what a business metric means. The v0.2 specification includes provenance, verification records, lifecycle status and a stale_after timestamp. It also defines attested computations: a concept that carries both the meaning of a value and the approved method to calculate it, so a consumer can check that a figure came from the validated calculation rather than from an agent that improvised it. And it explicitly leaves storage, serving and query infrastructure to the implementation. A field can describe when knowledge becomes stale. Something still has to detect source changes, refresh that knowledge and decide how consumers handle it. Likewise, a verification record describes a check; it does not enforce access permissions.

OKF can be part of a governed platform, including one backed by a database. The format and the operational platform have different responsibilities.

A personal second brain is a valid architecture

The famous LLM wiki pattern described by Andrej Karpathy is a useful example, and it is the pattern Google says OKF formalizes: keep source material, then let an LLM build and maintain linked Markdown pages around it.

I personally use Obsidian for my own notes, and I understand the appeal. Files are easy to inspect, easy to edit and easy to move. When I work alone, I can decide how to organize the corpus and review what the agent changes. With suitable backups and control over what reaches the model, this can be a perfectly reasonable architecture.

As the corpus grows, I can improve how I find information. Start with filenames, links, metadata and text search. I would recommend adding an index when repeated scans become inconvenient (Karpathy’s own note puts the useful limit of a plain index file at roughly a hundred sources and a few hundred pages, before a local search engine becomes worthwhile). SQLite FTS5 provides full-text search, while ordinary tables can hold document paths, tags, source revisions and indexing status. Markdown remains the authoritative content and SQLite holds a derived index that can be rebuilt from the files, as long as the indexing process detects modifications and deletions. With SQLite we have then included a database, even though there is no database server to operate. Note that the database capabilities can start inside a local application.

SQLite is also capable of serving multiple users through an application. Its deployment guidance highlights workload and concurrency as deciding factors: it permits one writer at a time per database file. The number of colleagues alone does not determine when PostgreSQL becomes necessary.

Shared memory becomes an application data problem

Strictly speaking, the LLM is a model inside an application (agents are programmatic). The surrounding application supplies tools, identity, retrieval and any state it needs to retain. An application can persist information without a database: files, object storage and durable logs are all possible storage choices. The precise requirement is that state needed after a restart or a new session must be stored somewhere outside the transient computation and loaded again when required.

What changes at team and organizational level is the set of guarantees we need around that state. A team could share reviewed documentation through Git and serve a mostly read-only corpus successfully. But consider a system where several agents update shared tasks, record approvals, remember customer-specific facts and act on behalf of users with different permissions. We now need conflict handling, authoritative versions, controlled visibility and recoverable history. Personal conventions are no longer enough to establish who may change what.

The comment thread under Karpathy’s gist is a good illustration of that evolution. By September, the people building their own versions of the pattern were reporting the same needs: write gates and proposal queues so that several clients can write without overwriting each other; provenance and verification stamps so that an unsupported answer filed into the wiki is not later treated as a source; staleness checks so that someone learns when a page stopped being true; and an MCP server in front of one canonical corpus so that different tools read and write the same memory. One contributor argued that a wiki needs to record both when a fact was true and when the system came to “believe” it, and that Markdown makes this hard. Those are database requirements, and the SQL standard has described that last one, bitemporal data, since 2011.

Four kinds of data, not one “memory”

“Memory” hides several responsibilities that do not need the same home:

Only the second kind cannot be rebuilt from something else. That is where I would normally put the database first, keeping knowledge documents where they are most useful. The database might already sit behind an existing service; it does not have to be a new PostgreSQL instance dedicated to the agent.
This is the argument I am making: when an organizational second brain starts coordinating work, it inherits the data-management requirements of a shared application. A database is often the most direct way to meet them. PostgreSQL being then my first choice for its maturity in both RDBMS and AI world as long as sovereignty advantages. This part of what we all call the vertical defensibility other and me included, talk about.

A larger context window does not replace data management

Could we skip all of this and simply give the model everything, in a larger context window?

Sometimes more context helps. If the task requires reasoning across an entire document, retrieving a few fragments may omit something essential. I would not claim that research has established a consensus against long context.

The LaRA benchmark, published at ICML 2025, found that the choice between retrieval and long-context processing depends on the model, task, context length and retrieval characteristics. An earlier EMNLP study found stronger average results for sufficiently resourced long-context processing on its benchmarks, with lower cost as an advantage of RAG. Neither establishes a universal winner. At the same time, Chroma’s context-rot report measured 18 models degrading as input length grows, often well before the window is full, so the window size is not a reliable guide to how much of it a model uses well.
What recent work does support is the need to manage context deliberately. Anthropic’s context-engineering guidance discusses selective retrieval, compaction and notes stored outside the active context. The Recursive Language Models paper, first published in December 2025, explores letting a model inspect and decompose external input programmatically. My understanding is that a larger context window is another capability to evaluate but it does not remove the need to select relevant information, track its origin or keep it current.

And even if a model could reliably process every document in the company, we would still have to decide which documents a particular user may access. More tokens do not define ownership, resolve conflicting updates or provide a recovery plan. Deeper thoughts on the matter here.

One operation, followed end to end

The shared state is where agents differ most from a retrieval demo, so let us follow one piece of it. An agent proposes a maintenance operation: rebuild an index on a busy table during tonight’s window. A person approves it. Another agent changes the plan. Execution times out. At each step, something must be recorded and something must be checked.

The proposal. The agent writes an operation record: the plan, a revision number starting at 1, and a state, awaiting approval. The plan can come from a runbook in Markdown; the operation itself is authoritative state, so it goes in the database.

The approval. A person approves revision 1. The approval row names the operation, the revision and the approver, and it is written by a trusted component that authenticated that person, never by the agent. An approved flag that the agent can set is not an approval.

The change. A second agent decides the window is too short and edits the plan. It read revision 1, so it updates on that condition:

UPDATE operation
   SET plan = $2,
       revision = revision + 1,
       state = 'awaiting_approval'
 WHERE id = $1
   AND revision = $3;  -- the revision this agent read

If someone changed the record in the meantime, the update touches zero rows and the agent has to read again instead of overwriting work it never saw. The edit also creates revision 2 and sends the operation back to awaiting approval: the person approved revision 1, not whatever the plan became afterwards. Depending on the operation, the same protection can come from a row lock or from serializable transactions with retry handling; PostgreSQL’s transaction isolation documentation explains why the isolation level matters.

The execution. Once revision 2 is approved, the execution service claims the operation in one statement, so two executors cannot both start it:

UPDATE operation o
   SET state = 'executing',
       attempt_id = $2
 WHERE o.id = $1
   AND o.state = 'approved'
   AND EXISTS (SELECT 1
                 FROM approval a
                WHERE a.operation_id = o.id
                  AND a.revision = o.revision)
RETURNING o.revision;

Zero rows means another executor got there first, or the approved revision is no longer the current one. The service then verifies its own authority to act and only then touches the target system. The attempt identifier is recorded before the action, so the action can be recognized later.

The timeout. The call to the target system times out. Maybe the rebuild never started, maybe it finished a second before the connection dropped. A database transaction cannot roll back an action performed outside it, and saving the workflow state does not make the external action atomic with the update that records its completion. So the state becomes requiring recovery, not failed and not completed. Reconciliation then asks the target: does the new index exist, and is it valid? Only then is the operation marked completed, or retried with the same attempt identifier so that a retry cannot apply the action twice. A blind retry is how an agent does the same thing twice.

The record. At the end, the database holds every revision, who approved which one, every attempt and its outcome. That history cannot be rebuilt from anything else, which is why it deserves a stricter recovery objective than the embeddings.

XState is one example of a library for expressing this logic: states, allowed transitions and guards that decide whether a transition may happen. It also saves and restores actor state, while the application supplies the persistent storage and the recovery design. A state machine helps express the process; its presence alone does not establish a security boundary. That said, I am quite fond of this project, and I think it is a significant piece of the puzzle. The responsibilities stay explicit:

The retrieval lab I describe below does not cover any of this. Safe autonomous writes and workflow recovery are a separate test, and the one I would want to see before an agent acts on production.

The other familiar questions

Can we recover it

For a DBA, storing data is only the beginning. We also need to know how much we could lose and how long recovery would take: our RPO and RTO. PostgreSQL provides WAL and the mechanisms needed for replication and recovery. Point-in-time recovery still requires a usable base backup and the required WAL. High availability requires a configured deployment and a tested failover process. Installing PostgreSQL does not give us all of this automatically. A replica is also not a substitute for a backup: it can faithfully reproduce an unwanted change, including one an agent made. An embedding index may be reproducible, but rebuilding it takes time and can incur embedding costs.

How do we know the knowledge is current

A source document changes. Its embedding still represents the previous version. Retrieval continues to work, but it works against outdated information. CDC can tell a pipeline that source data changed. It does not regenerate the embedding by itself, and it does not make an external embedding service part of the source transaction. My approach would be to track the source revision and the embedding revision explicitly, generate the replacement, verify that it still corresponds to the intended source revision, then publish the new content and embedding together. Deletion needs the same attention: removing a source row does not remove copies from caches, downstream stores or conversation history. In Embedding Versioning with pgvector, I argued that batch re-embedding is technical debt and event-driven refresh a precondition. For data that changes and drives decisions, I still think so. The more precise rule is that the freshness the use case requires decides whether batch is enough. A documentation corpus refreshed every night may meet its target; operational data behind a time-sensitive decision will not. Declare the target, then measure the delay, including failed refreshes and deletion propagation.

Who is allowed to retrieve it

PostgreSQL row-level security enforces row visibility in the engine, for every application and retrieval path that reads the same tables, as long as the application does not connect with a role that bypasses it. I built and measured this in AI agents on sensitive data: what PostgreSQL can enforce and what it can’t: synthetic banks, an agent answering questions over client documents, pgvector next to the data. Policies apply to the embedding tables as they do to the documents. The tenant is set inside every transaction, so it ends at commit and a pooled connection does not hand it to the next request, as long as nothing sets it at session level. But row visibility was one layer of six: column privileges, a register of what is sensitive, deterministic tokens, a gate on what leaves for the model provider, a separate vault server and the audit trail each carry part of the guarantee, and the guarantee ends where the labelling ends.
The database enforces the rule once the identity is set; it cannot tell whether the identity is the right one. The trusted service chooses the tenant from the authenticated user, and the agent must not be able to overwrite it, for example by running its own SQL on the same connection. On relevance, the measurements went against the usual fear: blunt redaction collapsed retrieval, while labelled tokens with entity-aware search matched raw text under the same search, the answers cited the expected documents as often, and zero detected sensitive values were sent to the model.

Can we reconstruct what happened

pgAudit records which statements ran and which objects they accessed; its optional row logging records a count, not the rows. It sees the database side of one step, not which question triggered it, what the model did with the rows, or whether the answer was right. For a RAG request I would also want a request identifier, the authenticated identity, the database role and whether it bypasses row-level security, the retrieved document identifiers and revisions, the policy version, the context actually sent, and the answer with its evaluation.
In the lab, I added Langfuse, self-hosted, to follow that path. Two lessons. The tracing layer is a database workload in its own right, PostgreSQL plus ClickHouse, to size, back up, secure and retain. And with evaluation scores next to the trace spans (a span is one recorded step, such as the search or the model call), a weak answer becomes a join: did the search miss the documents, or did the model ignore them?

It also stores every retrieved document once more. In the lab, a tool’s result was recorded when the tool returned, before the provider gate checked the next request, so a value the gate blocked could still reach the trace store. Tracing needs its own filtering boundary, applied before trace data leaves the application, rather than the assumption that the provider gate covers it. That variant is the next one I want to run.

Measure retrieval quality and security separately

Production RAG evaluation splits the retrieval step from the generation step. For retrieval, the classic measures, recall@k, precision@k and nDCG@k, still do the job (Understanding Confidence, Precision and nDCG defines them), and public benchmarks such as BEIR and MTEB still rank on nDCG@10. Their cost is a set of questions with the documents that should come back, written by someone who knows the data. Most tooling (Ragas, TruLens, DeepEval, the evaluators in Langfuse) adds measures scored by a second model acting as a judge: context precision and recall for the retrieval, faithfulness and answer relevance for the answer. Judges scale where human labels do not, but they make errors too, so I would calibrate them on a small labelled set. And each measure claims less than it seems: an answer that cites the expected documents has the right coverage, which is not proof that it is factually correct.
Two points matter for sensitive data. First, define the evaluation set per identity: the relevant documents that identity is permitted to retrieve. That avoids treating the exclusion of another customer’s data as a retrieval failure. Second, a judge model reads the question, the retrieved context and the answer, so it is one more place the data goes, like an external embedding service. Both belong in the same data-flow review as the model that answers. Measures computed from document identifiers need no judge at all.

Security testing needs its own checks. Does a request from tenant A ever return tenant B’s documents? Can a user change the retrieval identity? Do revoked permissions still allow access through a cache? Can retrieved content persuade an agent to use another tool with broader access? A high relevance score does not answer those questions. Unauthorized data can be extremely relevant to a query and still be forbidden. A mandatory access restriction is a requirement to satisfy; we can then work on preserving quality within it.

The operational work still needs an owner

A production agent depends on more than retrieval. It needs credentials, tool permissions, monitoring, deployment procedures and recovery paths, and approval before a consequential action, as in the example above. This is shared work between application teams, database teams, infrastructure and security. I would not call every one of these problems a database problem. Model reliability, prompt injection and the safety of external actions need their own controls; I discussed where the trust boundary sits for tools in RAG, MCP, Skills, and why governance changes everything. Database experience nevertheless covers a substantial part of the foundation.
Anthropic’s AI-native SDLC playbook gives a concrete example of this broader work. It describes versioned artifacts such as intent.md, spec.md and plan.md, alongside review, approval gates and operational feedback, and it distinguishes advisory instructions in skills from controls enforced through executable mechanisms such as hooks. Markdown stays valuable inside the workflow; the process around it establishes who may approve a change, what can execute and where the authoritative record lives. It is a vendor’s implementation guidance, so I would evaluate its controls against the actual environment rather than treat every example as a production guarantee.

Readiness needs evidence

Readiness can be described through evidence: a restore that met the recovery objective, a failover test that included the CDC consumer, a tenant-isolation test, a measured refresh delay, a retrieval evaluation based on the intended users and data, and, for an agent that acts, a test of the timeout above. We also need to know where the data goes. Keeping PostgreSQL in Switzerland does not keep the context sent to an external model there: embedding providers, inference endpoints, judges, traces, backups and support access all belong in the assessment, and the EU AI Act and the revised Swiss data-protection law both expect us to know it. How to judge that evidence against targets an organization declared for itself, rather than against someone else’s checklist, is the subject of Deciding in the Storm.

Conclusion

I see a useful opportunity here for database teams. We already know how to ask about consistency, permissions, recovery and operational ownership. Agentic applications give us another reason to ask those questions early.
A personal second brain can work well with files, search and an embedded database as needed. As it becomes a shared application, separate what it holds, define the guarantees its users depend on and choose the storage and workflow mechanisms that provide them. Keep formats such as OKF where portability and reviewable knowledge help.

Larger context windows give the model more room to work. The application still needs a reliable way to retain, update and govern the information it uses.

Related articles: AI agents on sensitive data: what PostgreSQL can enforce and what it can’t; RAG, MCP, Skills, and why governance changes everything; Embedding Versioning with pgvector; Understanding Confidence, Precision and nDCG; the Swiss PGDay 2026 talk “The cost of security debt in PostgreSQL when implementing AI workflows”.

L’article AI problems are database problems est apparu en premier sur dbi Blog.

Usually I see two answers for that. Keep AI away from that data, which in practice means there is no project. Or redact before sending: replace every name with [REDACTED] and send the rest. That one looks safe and fails quietly, because once every client is [REDACTED], the assistant can no longer tell one client from another because it is like joining two tables without a key. It answers, but without being relevant enough.

I built the layer in between, on PostgreSQL: row-level security, labels, the token columns and audit live in the database, the tokenizing and the gate in the application, the key and the mapping on a second server. Here is what it does, on a question from the lab (condensed from the run’s output):

question asked : What did Emmental Immobilien AG complain about regarding custody fees?
question sent  : What did CLIENT_43aa5d8366b5 complain about regarding custody fees?
model answer   : CLIENT_43aa5d8366b5 complained that the custody fees had increased without notice. [doc 10]
for a reviewer : Emmental Immobilien AG complained that the custody fees had increased without notice. [doc 10]

The model chose its own search, kept the token, answered correctly and cited its source. It never held the name. This is pseudonymization, not anonymization: the tokens are reversible by design (for everything downstream of the filter, only through a separate server), and the text around a token can still say a lot about someone. The goal is narrower and testable: no known sensitive value reaches the provider, and the answers stay as relevant as on raw text.

In my lab I take the example of a bank, because in Switzerland, sending client data covered by banking secrecy to an external API is a disclosure question under Article 47 of the Banking Act before it is a quality question, and FINMA expects banks to manage the data risks and third-party dependencies that come with AI. So the question is: can we build something secure enough to still use agentic capabilities? If it works in a sector as regulated as banking, most organizations can use it too. Retrieval techniques make governance possible; the other question is how far they go.
The lab has three synthetic banks, 240 clients, 18 relationship managers, 481 accounts, 30 servers and 1,521 free-text documents. It is small on purpose: big enough to measure, small enough to rebuild on a laptop and read every row. Every name in it is generated; where a generated company name turned out to match a firm in the Swiss commercial register, the generator swaps it, and any remaining resemblance is coincidental. This post walks through the data model, the vault that holds the tokens, the layers between the data and the model, a deliberate labelling gap and what catches it, the relevance measured with and without filtering, what a tracing tool ends up storing, and the query plans underneath.

The interesting part is not hiding the names. It is knowing precisely which data is sensitive: the labels are what stop the leak, the same labels are what give the relevance back, and a name that no label knows is exactly where the guarantee ends.

This demonstrates:

• Row-level security on the vectors, not only on the documents, with the tenant set per transaction.
• A label registry and a coverage query that fails toward a finding when a sensitive column is missing.
• Deterministic tokens computed with a keyed hash on a separate PostgreSQL server, which is the only way back for everything downstream of the filter.
• An egress gate that scans every request to the model, logs a hash of it, and blocks on a single finding.
• Relevance kept: with the same search and the same tools, answers on tokens are as good as answers on raw text.

Lab Environment Details

Bank server:
PostgreSQL 18.3, pgvector 0.8.6, pgAudit 18.0, postgresql_anonymizer 3.2.2.
Vault server:
PostgreSQL 18.3 with pgcrypto.
Embeddings computed locally on CPU:
dense voyage-4-nano at 1,024 dimensions, sparse SPLADE (prithivida/Splade_PP_en_v1) through fastembed.
Agent:
OpenAI gpt-6-luna through Chat Completions with function tools.
Tracing:
Langfuse 4.50 self-hosted from its official Docker Compose file (ClickHouse 25.12, PostgreSQL 17, Redis 7, MinIO), Python SDK 4.16. All in Docker on WSL2, in a 6 GB VM.

The lab is public and available with a runbook that replays every output below: lab 16 in pgvector_RAG_search_lab.
Building it takes about 40 minutes of embeddings on a CPU, and replaying every step about an hour and a half; the dry runs need no API key.

The data model

Two servers, three schemas. The bank server holds the business data, its filtered copies and the governance tables. The vault server holds only the key and the token mapping. No foreign key, foreign server or dblink crosses between them.

The bank schema: business data, its filtered copies, the mention index and the versioned embeddings. Each sensitive structured column sits next to its token column; the free text has redacted and tokenized copies (title copies omitted here).

Each document is stored three times, raw, redacted and tokenized. That is not how you would run production; it is how you measure what each choice costs, on the same rows, with the same model:

CREATE TABLE bank.documents (
    doc_id          int PRIMARY KEY,
    bank_id         text NOT NULL REFERENCES bank.banks,
    doc_type        text NOT NULL CHECK (doc_type IN ('advisor_note', 'incident', 'email')),
    title           text NOT NULL,
    body            text NOT NULL,
    created_at      timestamptz NOT NULL,
    title_redacted  text,
    body_redacted   text,
    title_tokenized text,
    body_tokenized  text,
    key_version     int
);

Embeddings are versioned: every row says which model, which text state and which vault key produced it. Why that matters, and the event-driven pipeline that keeps vectors current, is the subject of my embedding versioning post and its lab. A partial unique index allows at most one active version per state, so a new set can be built next to the old one and switched when it is ready:

CREATE UNIQUE INDEX embedding_versions_one_active ON bank.embedding_versions (state) WHERE is_active;

-- shortened: the repository version adds a hash of the embedded text
CREATE TABLE bank.embeddings (
    version_id int  NOT NULL REFERENCES bank.embedding_versions,
    doc_id     int  NOT NULL REFERENCES bank.documents,
    bank_id    text NOT NULL,
    dense      vector(1024) NOT NULL,
    sparse     sparsevec(30522) NOT NULL,
    PRIMARY KEY (version_id, doc_id)
);
CREATE INDEX embeddings_dense_hnsw ON bank.embeddings USING hnsw (dense vector_cosine_ops);
CREATE INDEX ON bank.embeddings (version_id, bank_id);

The bank_id on embeddings is deliberate. The vector table is the one people forget to protect: it gets the same tenant column, and the same row-level security policy, as the documents.

The governance schema gov is plain tables. A registry says which column is sensitive and where its token goes, and an egress log records every request that leaves, with a hash and the size of the payload and what the scanner found, never the payload itself. The gateway may read the log, insert into it and set each request’s outcome, nothing else:

CREATE TABLE gov.sensitive_columns (
    table_name   text NOT NULL,
    column_name  text NOT NULL,
    category     text NOT NULL,   -- CLIENT, PERSON, EMAIL, IBAN, HOST, IP, FREE_TEXT
    token_column text,            -- where the deterministic token is written
    PRIMARY KEY (table_name, column_name)
);

GRANT SELECT, INSERT ON gov.egress_log TO gateway;
GRANT UPDATE (outcome) ON gov.egress_log TO gateway;   -- the rest of the log is append-only

The vault, which everything else depends on

The key, and the mapping from a token back to its value, are what turn a token into a name. Anyone who only ever sees tokens (the model, the agent’s role, a tracing store) has no other way back, and a reviewer gets names only through it, so they live on a second PostgreSQL server, with its own roles and credentials, and no postgres_fdw or dblink on the bank side. The two containers also sit on separate Docker networks. On one laptop that is as far as it goes: the bank’s superuser can run programs on its server, and the vault’s port is still reachable through the Docker host. In production the vault belongs on its own host, behind a firewall that only lets the gateway in.

One honest limit of this lab’s layout: the bank tables keep each token next to its raw value (client_token beside client_name), because the pipeline needs both to filter the text. So the pipeline role and the bank’s administrators can map tokens back without the vault; what keeps the agent out is column privileges, below. One more route stays open in the lab: the agent’s role can read the raw-text embeddings kept for the measurement, and a sparse vector is indexed by words, so it partly exposes the words it was built from, names included. A stricter design moves the token columns out of the raw tables.

The vault server holds only the key and the token mapping. tokenizer can compute tokens but read neither table; reidentifier reads the mapping, never the key (the vault’s admin, a superuser in the lab, reads both).

Nobody reads the key, not even the role that creates tokens. Tokens are computed inside the vault by a SECURITY DEFINER function, which runs with its owner’s rights (in the lab the owner is the vault’s admin, a superuser, which a production setup would replace with a dedicated non-superuser owner), with a pinned search_path:

-- shortened: the repository version also raises when no key is active
CREATE FUNCTION vault.tokenize(p_category text, p_values text[])
RETURNS TABLE (value text, token text)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = vault, public, pg_temp
AS $$
#variable_conflict use_column
DECLARE
    k  bytea;
    kv int;
BEGIN
    SELECT secret, key_version INTO k, kv FROM vault.keys WHERE active;
    RETURN QUERY
    WITH src AS (
        SELECT DISTINCT v AS value FROM unnest(p_values) AS v WHERE v IS NOT NULL
    ), tok AS (
        SELECT s.value,
               upper(p_category) || '_' ||
               substr(encode(hmac(convert_to(upper(p_category) || ':' || lower(btrim(s.value)), 'UTF8'),
                                  k, 'sha256'), 'hex'), 1, 12) AS token
        FROM src s
    ), ins AS (
        INSERT INTO vault.mapping (token, key_version, category, value)
        SELECT DISTINCT ON (t.token) t.token, kv, upper(p_category), t.value
        FROM tok t ORDER BY t.token, t.value COLLATE "C"
        ON CONFLICT DO NOTHING
    )
    SELECT t.value, t.token FROM tok t;
END;
$$;

Functions are executable by PUBLIC by default, so the vault script revokes that before granting anything, and the public schema in that search_path is not writable by either role (the default since PostgreSQL 15, and revoked explicitly in the script):

REVOKE ALL ON ALL TABLES    IN SCHEMA vault FROM PUBLIC;
REVOKE ALL ON ALL FUNCTIONS IN SCHEMA vault FROM PUBLIC;
GRANT USAGE ON SCHEMA vault TO tokenizer, reidentifier;
GRANT EXECUTE ON FUNCTION vault.tokenize(text, text[]), vault.active_key_version() TO tokenizer;
GRANT SELECT ON vault.mapping TO reidentifier;
GRANT EXECUTE ON FUNCTION vault.active_key_version() TO reidentifier;

From the run, by role:

tokenizer:    SELECT * FROM vault.keys;
              ERROR:  permission denied for table keys
tokenizer:    SELECT * FROM vault.tokenize('client', ARRAY['Brenta Kontor AG', 'brenta kontor ag ']);
                     value       |        token
              -------------------+---------------------
               brenta kontor ag  | CLIENT_7119873645a3
               Brenta Kontor AG  | CLIENT_7119873645a3
reidentifier: SELECT * FROM vault.keys;
              ERROR:  permission denied for table keys

Three things the token is, and one it is not. It is the category plus the first 12 hex characters of an HMAC-SHA256 of the normalized value, a keyed hash nobody can reproduce without the key; whoever holds the tokenizer credential can still submit a guess and compare tokens (and the guess lands in the mapping), which is why that credential stays with the gateway. It is 48 bits, so by the birthday bound a collision within one category becomes a 1% risk around 2.4 million values; the largest category here has 481. And this version does not detect a collision: ON CONFLICT DO NOTHING would skip it silently, where a production version should raise.

What it is not is entity resolution. The hash input is category and normalized string, with no tenant and no entity ID, so two different clients with the same name would share a token, and the same name in two banks gets the same token. The synthetic data has no such pair, which is why the results below are not affected, but a real CRM has them. A production version would tokenize a tenant-scoped canonical ID, and map aliases to it explicitly. The lab does that in one place only: a server’s hostname and its full domain name share one token.

    hostname    |             fqdn              |    host_token     |    fqdn_token
----------------+-------------------------------+-------------------+-------------------
 alb-pg-prd-01  | alb-pg-prd-01.alder.internal  | HOST_f774b8916acd | HOST_f774b8916acd

Architecture: two servers, one way in

Where each layer sits: four on the bank server, the gate in the gateway process, the key and mapping on the vault server. The dotted paths go to the vault: tokens computed once and written back to the bank, and re-identification, used only for a reviewer.

One question end to end: the name never leaves the gateway and the model sees tokens only. The last two steps run only when a reviewer needs the real name; otherwise the answer stays in tokens.

The agent never writes SQL. It gets three tools, search_documents, client_profile and host_documents, each a plain SECURITY INVOKER function that runs as app_agent, so row-level security and column privileges apply inside it. In the terms of my RAG, MCP, Skills post, this is an agent with tool access, but every tool is a retrieval function under the database’s rules, so the DBA keeps the control, MCP and skills still being weaker on the governance aspects. app_agent has no privilege on any raw sensitive column (it reads tokens and attributes that do not identify anyone on their own, such as currency, domicile or balance), so a tool cannot return a client’s name even by mistake. Those attributes do reach the provider, tied to a token, which is part of why this is pseudonymization. (It can read the raw-text embeddings, which exist here only for the measurement; a production setup would not keep them.)

SELECT has_column_privilege('app_agent', 'bank.clients', 'client_name',  'SELECT') AS agent_reads_client_name,
       has_column_privilege('app_agent', 'bank.clients', 'client_token', 'SELECT') AS agent_reads_client_token;

 agent_reads_client_name | agent_reads_client_token
-------------------------+--------------------------
 f                       | t

Where the trust sits matters as much as the layers. The guarantee holds for what the agent can reach through those tools and that role. The gateway is the trusted part: it chooses the tenant, it holds the tokenizer and reidentifier credentials, and it decides what goes to the provider. It must derive the tenant from the authenticated user, never from an argument the model wrote, because row-level security enforces whatever bank the session declares without asking whether the session was entitled to declare it. A compromised gateway is outside what this design protects.

What PostgreSQL enforces, what it doesn’t, and why the ecosystem matters

The database enforces these by itself, on every query, whatever the application does: row-level security on the documents, the mention index and the vectors; column privileges that keep raw values away from the agent’s role; the revoked defaults and the SECURITY DEFINER function that keep the key unreadable in the vault; and pgAudit’s record of what the agent executed. It also holds the label catalog and reports what is missing from it through the coverage view, though a report only helps when someone reads it.

It does not do the rest, and I won’t pretend it does. The gate that scans every outbound request is a short Python function in the gateway. The tenant is chosen by the application. The labels are only as complete as the people who maintain them, and a name that is in no table passes. The tracing store and the model provider sit outside the database. And the vault is isolated by separate servers and networks (in the lab, separate containers on one host), not by SQL.

So why build it on PostgreSQL rather than around a separate vector store? Three reasons. First, one access model: the vectors live in the same engine as the rows they came from, under the same policy and the same privileges, so there is no second copy of every rule to keep in sync. Second, every database-side layer is core PostgreSQL or an extension, not a new product: pgvector for dense and sparse vectors, postgresql_anonymizer for the labels, pgcrypto for the keyed hash, pgAudit for the audit, all under the PostgreSQL License, maintained in the open and readable by the people who have to trust them. Third, everything except the chat model runs on your own machines: the two PostgreSQL servers, the embedding models (Apache 2.0, on CPU), and even the observability, since Langfuse (MIT, apart from its enterprise edition, the one its documentation lists retention under) runs on PostgreSQL and ClickHouse (Apache 2.0), plus a queue and an object store with their own licences. At run time the only external call in the lab is the chat model, and it is the call the gate scans, and in the governed setup blocks. For a bank that has to manage the third-party dependencies FINMA points to, that is what sovereignty means here: the model can move on premises without changing the data, the labels, the vault or the gate.

One design choice: entity first, then meaning

A hex string like CLIENT_43aa5d8366b5 means nothing to a semantic model, so dense search barely uses it. But the labels give me something better than semantics: while filtering the text, the tokenizer records which document mentions which entity, in bank.document_mentions. When a question names an entity, the search starts with the documents that mention it, ranks those by meaning, then adds the rest:

-- excerpt of bank.retrieve()
ent AS (
    -- documents mentioning every entity the question names
    SELECT m.doc_id FROM bank.document_mentions m
    WHERE p_method = 'entity' AND cardinality(p_tokens) > 0 AND m.token = ANY(p_tokens)
    GROUP BY m.doc_id HAVING count(DISTINCT m.token) = cardinality(p_tokens)
), ent_scored AS (
    SELECT e.doc_id,
           1.0 + 1.0 / (60 + row_number() OVER (ORDER BY e.dense <=> p_dense))
               + 1.0 / (60 + row_number() OVER (ORDER BY e.sparse <#> p_sparse)) AS score
    FROM bank.embeddings e JOIN v USING (version_id)
    WHERE e.doc_id IN (SELECT doc_id FROM ent)
)

Inside the entity’s documents the order is reciprocal rank fusion of dense and sparse, the same as everywhere else. The 1.0 + puts them ahead of everything that does not mention the entity. It is a choice, not a constant: if your questions often name an entity only in passing, you would add a boost instead of ranking first. The mention index is a plain table with a B-tree on token, under the same row-level security as the rest. Raw text can use it too, which matters for a fair comparison below. Run on its own, as app_agent with bank_a set (inside the function it is a CTE, and the function’s own plan is a single opaque function scan):

EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
SELECT m.doc_id FROM bank.document_mentions m
WHERE m.token = ANY(ARRAY['CLIENT_43aa5d8366b5'])
GROUP BY m.doc_id HAVING count(DISTINCT m.token) = 1;

 GroupAggregate (actual time=0.072..0.075 rows=6.00 loops=1)
   Group Key: doc_id
   Filter: (count(DISTINCT token) = 1)
   ->  Sort (actual time=0.066..0.067 rows=6.00 loops=1)
         ->  Bitmap Heap Scan on document_mentions m (actual time=0.042..0.044 rows=6.00 loops=1)
               Recheck Cond: (token = ANY ('{CLIENT_43aa5d8366b5}'::text[]))
               Filter: (bank_id = current_setting('app.bank_id'::text, true))
               ->  Bitmap Index Scan on document_mentions_token_idx (actual time=0.025..0.025 rows=6.00 loops=1)
                     Index Cond: (token = ANY ('{CLIENT_43aa5d8366b5}'::text[]))
 Execution Time: 0.185 ms

(Shortened.) The token lookup is one index probe and six rows. Note where the row-level security predicate lands: here it is a Filter after the lookup, because the B-tree is on token alone. Keep that in mind for the vector plan below.

Row-level security on the vectors

Every business table that carries bank_id gets the same policy, the vector table included:

ALTER TABLE bank.embeddings ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON bank.embeddings
    USING (bank_id = current_setting('app.bank_id', true));

The bank is set per transaction with set_config(..., true), the function form of SET LOCAL, so a transaction-mode pooler cannot hand one bank’s setting to the next request. As app_agent:

bank=> SELECT count(*) AS documents_without_tenant FROM bank.documents;
 documents_without_tenant
--------------------------
                        0

bank=> BEGIN;
bank=> SELECT set_config('app.bank_id', 'bank_a', true);
bank=> SELECT bank_id, count(*) AS documents FROM bank.documents GROUP BY bank_id;
 bank_id | documents
---------+-----------
 bank_a  |       508
bank=> COMMIT;

bank=> SELECT count(*) AS documents_after_commit FROM bank.documents;
 documents_after_commit
------------------------
                      0

Be precise about why both counts are zero, because they are not the same case. On PostgreSQL 18.3, current_setting('app.bank_id', true) returns NULL when the setting was never defined in the session, and an empty string once an earlier transaction has set it locally: the custom setting now exists, with an empty value. Neither matches a bank_id, so a session that forgets its bank gets nothing. If you would rather have one case, NULLIF(current_setting('app.bank_id', true), '') in the policy makes both NULL. And a transaction-local value only overrides: if something set the parameter at session level earlier, that value is back after the commit, which is why the pooling argument assumes connections that never set it at session level.

The raw client name is refused before row-level security even matters: SELECT client_name FROM bank.clients as app_agent fails with permission denied. Neither role owns the tables; if your application connects as the table owner, add FORCE ROW LEVEL SECURITY, or the owner sees every row. The pipeline role, gateway, has BYPASSRLS because it serves all three banks, and the agent never gets near it.

Labelling, the gap, and what caught it

The registry says what is sensitive; a procedure turns it into security labels, with postgresql_anonymizer as the label provider, so the catalog (pg_seclabels) is the one place to ask whether a column is labelled. The coverage view asks it for every text column of the business tables:

-- shortened: the repository version also skips three technical tables and the
-- derived columns (tokens, filtered copies, text hashes)
CREATE VIEW gov.label_coverage AS
SELECT c.table_name, c.column_name, sl.label, sc.category,
       (sl.label IS NOT NULL) AS labelled
FROM information_schema.columns c
LEFT JOIN pg_seclabels sl
       ON sl.provider = 'anon' AND sl.objtype = 'column'
      AND sl.objoid = format('bank.%I', c.table_name)::regclass
      AND sl.objsubid = c.ordinal_position
LEFT JOIN gov.sensitive_columns sc
       ON sc.table_name = c.table_name AND sc.column_name = c.column_name
WHERE c.table_schema = 'bank' AND c.data_type = 'text'
  AND c.column_name NOT IN ('bank_id', 'doc_type', 'client_type', 'currency',
                            'role', 'environment', 'domicile', 'category', 'token');

The choice that matters is the direction of the list: it excludes what is known to be harmless, so a new text column shows up as a gap until someone labels it or excludes it on purpose. It fails toward a finding, not toward silence. It only sees text columns; a sensitive value stored as inet, date or jsonb needs its own rule. I left one sensitive column out on purpose, the clients’ phone numbers:

bank=# SELECT table_name, column_name, category, labelled FROM gov.label_coverage ORDER BY labelled, table_name;
      table_name       |  column_name  | category  | labelled
-----------------------+---------------+-----------+----------
 clients               | contact_phone |           | f
 accounts              | iban          | IBAN      | t
 clients               | client_name   | CLIENT    | t
 ...
(10 rows)

Tokenizing then replaces every labelled value in the free text, using a dictionary of every value the labelled columns hold, matched as whole words ignoring case, longest first, with the grouped spelling of each IBAN added. One caveat before the numbers: in this lab the dictionary and the scanner read hand-written lists of columns that match the registry, rather than reading the registry itself. A production version should derive both from it.

Building the governed corpus: the registry drives the labels and the coverage check, the vault computes the tokens, and one dictionary pass filters the text, fills the mention index and feeds the local embedding models.

$ python python/scan_corpus.py
raw         7171 values in 1521 documents: CLIENT 3024, PERSON 1833, EMAIL 960, HOST 630, IBAN 480, PHONE 154, IP 90
redacted     154 values in  154 documents: PHONE 154
tokenized    154 values in  154 documents: PHONE 154

The raw corpus carries 7,171 detected occurrences of sensitive values. After tokenization what is left is exactly the gap: 154 phone numbers from the column nobody labelled. Now the governed agent, asked about a client whose note carries that phone number, in a dry run that builds the real request and stops before it leaves:

$ python python/agent.py --dry-run --bank bank_a \
    "What did Emmental Immobilien AG complain about regarding custody fees?"
question sent : What did CLIENT_43aa5d8366b5 complain about regarding custody fees?
  [tool] search_documents returned:
    doc 10 advisor_note: PERSON_65a94b3e0336 met CLIENT_43aa5d8366b5. The client complained that the custody fees rose without notice. Next step agreed with CLIENT_43aa5d8366b
    doc 12 advisor_note: PERSON_65a94b3e0336 met CLIENT_43aa5d8366b5. The client complained that the custody fees rose without notice. Next step agreed with CLIENT_43aa5d8366b
    doc 11 email: From EMAIL_288e9f2f11a1. Dear client, following our call, CLIENT_43aa5d8366b5 complained that the custody fees rose without notice. Please confirm the
    doc 14 email: From EMAIL_288e9f2f11a1. Dear client, following our call, CLIENT_43aa5d8366b5 requested an ESG screening of current equity holdings. Please confirm th
    doc 13 advisor_note: PERSON_65a94b3e0336 met CLIENT_43aa5d8366b5. The client requested an ESG screening of current equity holdings. Next step agreed with CLIENT_43aa5d8366
  [egress] 3729 chars, filtering=tokenized, scanner hits=1 ['PHONE']  BLOCKED
  stopped at the egress gate: 1 sensitive value(s) in outbound payload: ['PHONE']

Every retrieved document is the right client and every name, email and account is a token, but doc 12 still carries the phone number. The gate stopped the request: that is what prevented the transmission. It caught the number because the scanner’s own list includes the phone column (I left it out of the registry on purpose, so I knew), and a Swiss phone-number pattern would have matched it anyway. Then the label fixed the filtering:

bank=# INSERT INTO gov.sensitive_columns VALUES ('clients', 'contact_phone', 'PHONE', 'phone_token');
bank=# CALL gov.apply_labels();
$ python python/tokenize_corpus.py
...
$ python python/agent.py --dry-run --bank bank_a "What did Emmental Immobilien AG complain about regarding custody fees?"
  ... (same question sent, same five documents)
  [egress] 3731 chars, filtering=tokenized, scanner hits=0
  payload cleared the gate (dry run: not sent)

Same documents, two characters longer (a 16-character phone number became an 18-character token), zero detected values. The gate stops what it recognizes; the label makes the filter remove it before it ever gets there. Both are needed, and neither knows a name that is in no table: nine documents each name a lawyer who exists in no column (Hugo Vogel in one, Selina Python in another), and those nine pass. Deterministic filtering is exactly as good as your labelling. Named-entity recognition can be added as a second net, but it puts a probabilistic step back into the path of a guarantee; I would rather know the ceiling and work on coverage.

Did filtering cost relevance?

Retrieval

36 labelled questions, 30 naming a client or a server and 6 generic, run against the raw, redacted and tokenized embeddings as app_agent with its bank set:

$ python python/measure.py
state      method  set       recall@5  nDCG@5  recall@10  nDCG@10
raw        hybrid  entity       0.956   0.953      1.000    0.973
raw        entity  entity       1.000   0.991      1.000    0.991
redacted   hybrid  entity       0.192   0.185      0.408    0.284
tokenized  dense   entity       0.431   0.447      0.625    0.541
tokenized  sparse  entity       0.822   0.827      0.897    0.860
tokenized  hybrid  entity       0.650   0.670      0.775    0.731
tokenized  entity  entity       1.000   0.990      1.000    0.990

(Shortened; the six generic questions score 1.000 everywhere, which says they are too easy. measure.py prints the full table.) Redaction destroys entity questions, 0.192, because the documents no longer say who. Opaque tokens alone recover part of it: the dense model barely uses a hex string (0.431), the sparse model does (0.822). Entity-aware search over the mention index brings tokenized retrieval to 1.000, the same as raw with the same search. Redaction cannot use it: a redacted question names no one.

Answers

Retrieval is not what a relationship manager reads, so the model answered all 36 questions, citing documents as [doc N], with deterministic scoring and no model judging another model. To isolate the cost of the tokens, the two setups compared here differ in the text and almost nothing else: both use the same entity-aware search, under the same row-level security and candidate limits, and the same single search tool. One reads raw text, the other tokens. (Two small differences remain: the entity is found by its token on one side and by its name on the other, and the raw side’s requests are logged but never blocked by the gate; none was blocked on either side in this run.) Both ran together on 3 October:

$ python python/evaluate_answers.py --label fair2 --modes raw_entity,tokenized_search
mode       set      cited rec cited prec context rec expl. entity blocked
raw_entity entity       0.722      0.989       1.000          1.0       0
tokenized_search entity       0.719      0.993       1.000          1.0       0
egress: {'tokenized_search': {'sent': 74, 'values_sent': 0, ...}, 'raw_entity': {'sent': 73, 'values_sent': 871, ...}}

(Shortened to the entity questions; the generic ones score 1.000 in both.) Both setups had every expected document in front of the model (context recall 1.000), and both cited the same share of them, 0.722 against 0.719. With the same search, the tokens cost nothing measurable. What changed is what left: 871 detected occurrences of sensitive values for the raw setup, none for the tokenized one.

For reference, the naive setup (raw text, plain hybrid search across all three banks, filtered to one bank afterwards) put fewer expected documents in front of the model, context recall 0.981, and sent 1,134 detected occurrences in 75 requests, in a separate run the same day. The naive setup differs from the entity-aware ones in two ways at once, the search method and searching all banks before filtering, so these runs cannot say which one accounts for the gap; what it does show is that the tokens are not the cost.

Cited recall counts expected documents cited, which is document coverage, not answer completeness: when three notes repeat the same complaint, citing one of them can be a complete answer. So the honest claim is narrower than “better answers”: on this corpus, filtering keeps the answers grounded in the right documents while nothing known leaves.

What a tracing tool keeps

The egress log tells me what left and how big it was, and pgAudit tells me what app_agent executed. Neither tells the story of one question: what was asked, which documents came back, under which role, what the model answered, and whether it was right. So I traced every question with Langfuse, the open-source tracing and evaluation tool for LLM applications that ClickHouse acquired in January 2026, self-hosted next to the lab. Look first at what it runs on: PostgreSQL for its transactional data, ClickHouse for traces and scores, Redis as a queue, S3-compatible storage for the raw events. AI observability is a database workload.

Langfuse 4.50, self-hosted: one trace per question, marked off for the naive setup and tokenized for the governed one in the metadata. The naive rows show client names in the input and output columns; the governed rows show tokens.

Each retrieval step records the database role, whether that role bypasses row-level security, and the document IDs, so the role question becomes SQL on the ClickHouse table that holds the spans, here on the traced run of 2 October:

clickhouse> SELECT tags[1] AS mode,
                   metadata_values[indexOf(metadata_names, 'db_role')]    AS db_role,
                   metadata_values[indexOf(metadata_names, 'bypass_rls')] AS bypass_rls,
                   metadata_values[indexOf(metadata_names, 'doc_ids')]    AS doc_ids
            FROM events_full FINAL
            WHERE session_id = 'langfuse' AND name = 'search_documents' AND input LIKE '%succession%'
            ORDER BY mode, start_time LIMIT 1 BY mode;
   ┌─mode──────┬─db_role───┬─bypass_rls─┬─doc_ids────────────────┐
1. │ off       │ gateway   │ true       │ [18, 19, 206, 100, 22] │
2. │ tokenized │ app_agent │ false      │ [18, 19, 16, 17, 292]  │
   └───────────┴───────────┴────────────┴────────────────────────┘

The naive search ran as a role that bypasses row-level security and stayed inside its bank only because the application’s query filtered on it. And because the evaluation’s scores sit next to the spans, a quality question becomes a join: for the governed answers that cited fewer expected documents, what had the search returned? For one of the two worst in that run, a question about replication lag on a server, the expected documents were 948 to 951; the search returned all four at the top, and the model cited one. That is not a retrieval problem, and without the trace next to the score I would have gone tuning the index.

Then I pointed the egress scanner at the trace store itself, for the three-setup run of 3 October:

$ python python/scan_traces.py --label fair
ClickHouse, table events_full, session fair
  mode       traces  spans  spans with values  sensitive values
  off            36    150                144              2336
  raw_entity     36    144                138              1850
  tokenized      36    170                  0                 0
object storage, bucket langfuse/events/otel, session fair
  (same counts, shortened)

Tracing records what the application sends and receives; that is its job. On the naive path the trace store ends up with 2,336 detected occurrences for questions that sent 1,134 to the provider: every retrieved document is stored once more as the search result, on top of the copies inside each model call, plus the questions and answers. And all of it twice, in ClickHouse and in object storage. On the governed path both hold zero detected values, and the reason is precise: the search result is recorded when the tool returns, before the gate scans the next request, so the traces are clean only because the tools return tokens. While the phone gap was open, doc 12’s number would have reached the trace store even though the gate blocked the request.

The naive setup, as Langfuse stored it (2 October): the search step’s output holds the raw text, with the client’s name and the advisor’s name and email in clear, and an IBAN and a phone number too, masked here for publication.

The governed setup, same kind of step: every name in the stored search result is a token.

So the observability layer is one more copy of the data. In the Langfuse 4.50 I ran, the table holding the spans carries a full-text index on every span’s input and output, so questions, retrieved documents and answers are searchable word by word. On a self-hosted instance that copy is kept indefinitely unless you configure retention, a setting its documentation lists only under the Enterprise Edition. The SDK has masking hooks that run before trace data leaves the application; that is where I would plug the gate’s scanner, so the trace store can never hold more than the provider was allowed to see. I have not run that variant yet.

What the plans show

The dense path as app_agent with bank_a set, using a stored vector as the query. The query has no bank predicate:

EXPLAIN (ANALYZE, BUFFERS, COSTS OFF)
SELECT e.doc_id FROM bank.embeddings e
JOIN bank.embedding_versions v USING (version_id)
WHERE v.state = 'tokenized' AND v.is_active
ORDER BY e.dense <=> :'qd'::vector LIMIT 10;

 Limit (actual time=1.936..1.938 rows=10.00 loops=1)
   Buffers: shared hit=1762
   ->  Sort (actual time=1.935..1.936 rows=10.00 loops=1)
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Nested Loop (actual time=0.051..1.850 rows=508.00 loops=1)
               Buffers: shared hit=1759
               ->  Index Scan using embedding_versions_one_active on embedding_versions v (actual time=0.006..0.006 rows=1.00 loops=1)
                     Index Cond: (state = 'tokenized'::text)
               ->  Index Scan using embeddings_version_id_bank_id_idx on embeddings e (actual time=0.019..0.144 rows=508.00 loops=1)
                     Index Cond: ((version_id = v.version_id) AND (bank_id = current_setting('app.bank_id'::text, true)))
                     Buffers: shared hit=64
 Execution Time: 2.234 ms

(Shortened: sort key, index searches and planning lines removed.)

Reading it from the bottom: the active version comes from the partial unique index, then an index scan on (version_id, bank_id) returns the 508 rows of one bank and one version in 64 buffers, and a top-N heapsort scores all of them exactly and keeps ten. At this size the planner does not use the HNSW index at all.

Two details are worth the read. The bank_id = current_setting(...) condition is not in the query: it comes from the policy, and because the index is on (version_id, bank_id), the planner used it as an index condition next to the version. On the mention index above, with no bank_id in the B-tree, the same predicate was a filter after the lookup: where row-level security lands in a plan depends on your indexes. And of the 1,762 buffers, the index scan accounts for 64; the rest is reading the vectors back. pgvector stores vector with the EXTERNAL storage mode, and at about 4 KB each (1,024 floats plus a small header) the dense vectors live out of line: 48 MB of TOAST against 8 MB of heap, for 9,126 rows across the six embedding versions the table holds. The sparse vectors average 773 bytes and stay inline, and the same plan on the sparse path reads 66 buffers. Over five repeated runs the dense path took 1.9 to 4.5 ms, the sparse path under 1 ms, and the hybrid function 5 to 14 ms.

With much larger tables per bank the planner may switch to the HNSW index, and a plain HNSW scan stops after ef_search candidates, some of which row-level security and the version filter then remove, since one HNSW index spans every version. That is why bank.retrieve() carries SET hnsw.iterative_scan = strict_order and SET hnsw.ef_search = 100 in its definition: the index keeps scanning until enough rows pass the filter or its configured scan limits are reached, and the settings cannot leak into the caller’s session. I did not measure where that switch happens.

Underneath, there is nothing exotic. Tenant, version and entity are predicates on ordinary indexes, the vault is another PostgreSQL server with revoked defaults, the gate is a short function writing to a table, and the vectors are TOASTed values like any other. Every layer is something a DBA already knows how to operate, back up and audit.

A note on method

Everything above was captured from runs of the lab on 1, 2 and 3 October 2026, command by command, and the runbook in the repository replays it, including the two comparison runs (step 19). The data is synthetic and generated with a fixed seed; vault tokens differ per vault (different hex, same behaviour). Embeddings ran on a CPU, between about 0.7 and 3.5 documents per second depending on the run, which is why the corpus is 1,521 documents and not a million.

The question set is small and generous: each entity question names one entity, and the generic ones are too easy. An outside review of an earlier run found questions whose wording matched only one of the ways their documents phrase the topic, which rewarded a correct “nothing found”; I fixed the wording and every number above uses the corrected set. Answers come from a model and vary between runs, which is why the compared setups were always run together. These numbers show the mechanism on this corpus; they are not a production benchmark.

“Zero” in this post always means zero detected occurrences: zero matches against the raw values of the bank’s sensitive columns and a few patterns, which a name in no table escapes. The scanner reads the payload with accents intact, so a name like Müller is matched too, though the lab data itself is plain ASCII. An outcome of sent in the egress log means the provider accepted the request; a request logged as failed may still have reached it before the error, so the log proves what was sent, not everything that was attempted. Key rotation is scripted but I have not run it end to end, and re-tokenizing overwrites the tokenized text in place, so older vector versions are not a full rollback. The agent ran on gpt-6-luna with reasoning turned off: in my runs, Chat Completions refused function tools for that model unless it was.

Takeaways

I often hear that you cannot keep sensitive data out of an AI pipeline and still get useful answers. On this lab you can, with one condition: labelling. Row-level security on the vectors decides which rows a session sees (which, as I showed in my Swiss PGDay talk, also improves precision), the labels decide what is sensitive, deterministic tokens replace it the same way every time, the gate proves what left, and the vault keeps the way back, for everything downstream of the filter, on another server. With the same search and tools, the answers on tokens were as good as the answers on raw text, and nothing known left. And the tool you add to watch all of it becomes a copy of the data unless what it records is filtered too.

This transfers to any assistant over CRM notes, a ticketing tool or a CMDB, wherever the useful text names people and systems, and it does not depend on which model you use or where it runs. Its weak points are said above: names that live in no table, a tracing store that sits outside the gate unless you put it behind, and a token that matches strings, not identities. It also makes the token mapping sensitive in its own, a secret with a required life cycle somehow.
A stale mapping increases the risk of a leak; a rotated one has to keep its history, for traceability and for questions that span a period with several mapping versions. How long to keep which version, and who may read it, is a question the business has to answer, and the real answer is probably a dynamic mapping system. This lab shows that the approach is possible, not how to run it in production.

Agentic projects put pressure on governance at a level organizations have never known, and it is understandable to feel overwhelmed, even afraid. To me this is an engineering problem, and we already hold most of the keys; the challenge is that the storm we are living through makes it hard to see clearly. Observability and open source will prevail !

L’article AI agents on sensitive data: what PostgreSQL can enforce, and what it can’t est apparu en premier sur dbi Blog.

Can this also be applied to PostgreSQL?

In short, yes, it is possible. Let’s see how.

Here is the list of the previous blog posts:

• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-2-3/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-rest-api-with-sql-server-2025-3-3/

Partitioning and filesystem

We will reuse the sqlpool ZFS pool created in the first part of this series.

We identify the 300 GB disk attached to the VM. In our case, it is /dev/sdb, backed by the sqlpool/pve/vm-307-disk-0 zvol on the Proxmox side:

lsblk

We create a single partition of type Linux filesystem:

sudo sgdisk -n 1:0:0 -t 1:8300 /dev/sdb

We format the partition with XFS, which is the most commonly recommended filesystem for PostgreSQL data directories:

sudo mkfs.xfs -L pgdata /dev/sdb1 -f

We verify the result:

ahi@pgl:~$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="pgdata" UUID="028afa2f-7bb3-4a40-92aa-91c1a33f18ae9" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="bd54f285-092e-4b1b-ba5e-6877f054fa7"

Mountpoint:

We create the mount point:

sudo mkdir -p /pgdata

Persistent mount via fstab:

We add the mount entry to /etc/fstab using the filesystem label rather than the device name. The device name (/dev/sdb) may change if disks are added or removed while the label remains stable:

echo 'LABEL=pgdata /pgdata xfs noatime,nodiratime 0 2' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /pgdata

We verify that the volume is mounted:

df -h /pgdata

PostgreSQL installation

We install PostgreSQL 18 from the official PGDG repository, which provides the latest PostgreSQL versions for Ubuntu:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt install -y postgresql-18

Cluster creation on /pgdata:

The Ubuntu packages create a default cluster under /var/lib/postgresql. This is not what we want. The data files and the WAL must both reside on the ZFS-backed volume, so that a single ZFS snapshot captures a consistent state of the database. If they were on different volumes, the snapshot would not be atomic.

We drop the default cluster and recreate it on /pgdata:

ahi@pgl:~$ sudo pg_dropcluster --stop 18 main
sudo install -d -o postgres -g postgres -m 700 /pgdata/18
sudo pg_createcluster -d /pgdata/18/main 18 main
sudo systemctl enable --now postgresql@18-main
Creating new PostgreSQL cluster 18/main ...
/usr/lib/postgresql/18/bin/initdb -D /pgdata/18/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /pgdata/18/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
             Ver Cluster Port Status Owner     Data directory  Log file
             18  main    5432 down   postgres  /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql@18-main.service → /usr/lib/systemd/system/postgresql@.service.

We verify that the cluster is online and located on the right volume:

ahi@pgl:~$ pg_lsclusters
Ver Cluster Port Status Owner     Data directory  Log file
18  main    5432 online <unknown> /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
ahi@pgl:~$ sudo -u postgres psql -c "SHOW data_directory;"
 data_directory
-----------------
 /pgdata/18/main
(1 row)

ahi@pgl:~$ sudo -u postgres psql -c "SELECT version();"
                                                             version
-------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
(1 row)

We can also confirm that the WAL directory lives inside the data directory, and therefore on the zvol:

ls -ld /pgdata/18/main/pg_wal

Creating a large database

We need a database large enough to make traditional backup and restore operations time-consuming. In the SQL Server part of this series, we used the StackOverflow database (about 207 GB). For PostgreSQL, we use pgbench, the benchmarking tool shipped with PostgreSQL.

We create the database and initialize it with a scale factor of 10000. This produces a database of approximately 146 GB, with 1 billion rows in the pgbench_accounts table:

sudo -u postgres createdb bench
sudo -u postgres pgbench -i -s 10000 --partitions=8 bench

A few minutes later:

We can monitor the data growth during the initialization:

watch -n 30 'df -h /pgdata'

A few minutes later:

On the Proxmox side:

After some time, the process completes:

vacuuming...

creating primary keys...
done in 1559.55 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 598.12 s, vacuum 675.53 s, primary keys 285.88 s).
ahi@pgl:~$

We check the database size:

We run a checkpoint before taking the snapshot. The recovery process starts replaying the WAL from the last checkpoint. By running it right before the snapshot, almost nothing needs to be replayed when the database starts after a restore:

sudo -u postgres psql -c "CHECKPOINT;"

Comparison with SQL Server:

On the SQL Server side, we had to run SUSPEND_FOR_SNAPSHOT_BACKUP and BACKUP WITH METADATA_ONLY. On the PostgreSQL side, none of that is needed.

The data files and the WAL are on the same zvol. An atomic ZFS snapshot therefore captures a state equivalent to a power loss, and PostgreSQL is designed to recover cleanly from that state through crash recovery: the WAL is replayed from the last checkpoint. This is documented and officially supported.

The snapshot is the backup. There is no .bkm file, no metadata backup.

Snapshot process flow

On the Proxmox side, we create the snapshot and protect it with a hold:

SNAP="sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)" 
zfs snapshot "$SNAP" 
zfs hold sqlsnap "$SNAP" 
echo "$SNAP"

The hold protects the snapshot from an accidental destruction, as we did in part 2 of this series. We note the exact snapshot name, it will be needed for the restore.

The database stays online during the whole operation. No I/O freeze is required.

We list the snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We drop the database then we restore the snapshot:

We run the snapshot restore procedure. On the VM, we stop the cluster and unmount the volume:

sudo systemctl stop postgresql@18-main
sudo umount /pgdata

On the Proxmox side, we want to restore our snapshot. We can list the available snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We roll back the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T165409

On the VM, we mount the volume and start the service:

sudo mount /pgdata
sudo systemctl start postgresql@18-main

We check a few elements in the logs:

sudo tail -30 /var/log/postgresql/postgresql-18-main.log

The service shutdown, then the restart:

UTC [232108] LOG:  database system is shut down
UTC [233968] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
UTC [233968] LOG:  listening on IPv4 address "0.0.0.0", port 5432
UTC [233968] LOG:  listening on IPv6 address "::", port 5432
UTC [233968] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

PostgreSQL detects that the database was not shut down properly and replays the WAL. This is the same crash recovery mechanism as in SQL Server. Finally, the database starts.

UTC [233974] LOG:  database system was not properly shut down; automatic recovery in progress
UTC [233974] LOG:  redo starts at 20/2CB76278
UTC [233974] LOG:  invalid record length at 20/2CB76380: expected at least 24, got 0
UTC [233974] LOG:  redo done at 20/2CB76348 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
UTC [233974] LOG:  checkpoint starting: end-of-recovery immediate wait
UTC [233972] LOG:  checkpoint complete: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.009 s, total=0.030 s; sync files=0, longest=0.000 s, average=0.005 s; distance=0 kB, estimate=0 kB; lsn=20/2CB76380, redo lsn=20/2CB76380
UTC [233968] LOG:  database system is ready to accept connections

We then verify that the database is available again:

sudo -u postgres psql -d bench -c "SELECT pg_size_pretty(pg_database_size('bench'));"

Consistency proof under load

The previous test was done on a quiesced database: we ran a CHECKPOINT right before the snapshot, and nothing was writing. The real question is: what happens if the snapshot is taken while the database is being written to?

This is where PostgreSQL differs the most from SQL Server. There is no SUSPEND_FOR_SNAPSHOT_BACKUP. We take the snapshot in the middle of the write activity and we let the WAL replay do the work.

We start the load. The built-in pgbench script runs a TPC-B-like transaction: three UPDATE statements on the accounts, tellers and branches tables and one INSERT into the history table:

sudo -u postgres pgbench -c 8 -j 4 -T 300 bench &

While the load is running, we take a snapshot on the Proxmox side:

zfs snapshot sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)

zfs hold sqlsnap sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551

No CHECKPOINT this time, no freeze. The database is actively writing while the snapshot is taken.

The state after some time under load:

We stop the service:

sudo systemctl stop postgresql@18-main 
sudo umount /pgdata

We restore the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551 

We mount the volume, start the service and check the logs:

sudo mount /pgdata 
sudo systemctl start postgresql@18-main

This time the log shows a real recovery:

2026-08-03 20:49:03.244 UTC [234405] LOG:  database system is shut down
2026-08-03 20:50:34.333 UTC [234620] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv6 address "::", port 5432
2026-08-03 20:50:34.335 UTC [234620] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2026-08-03 20:50:34.343 UTC [234626] LOG:  database system was interrupted; last known up at 2026-08-03 20:44:36 UTC
2026-08-03 20:50:34.381 UTC [234626] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-03 20:50:34.384 UTC [234626] LOG:  redo starts at 20/7A3618B0
2026-08-03 20:50:38.024 UTC [234626] LOG:  invalid record length at 20/8BC0A4F8: expected at least 24, got 0
2026-08-03 20:50:38.024 UTC [234626] LOG:  redo done at 20/8BC0A4D0 system usage: CPU: user: 0.71 s, system: 0.63 s, elapsed: 3.63 s
2026-08-03 20:50:38.028 UTC [234624] LOG:  checkpoint starting: end-of-recovery immediate wait
2026-08-03 20:50:53.936 UTC [234624] LOG:  checkpoint complete: wrote 105355 buffers (53.6%), wrote 5 SLRU buffers; 0 WAL file(s) added, 17 removed, 0 recycled; write=3.648 s, sync=12.232 s, total=15.911 s; sync files=189, longest=12.223 s, average=0.065 s; distance=287395 kB, estimate=287395 kB; lsn=20/8BC0A4F8, redo lsn=20/8BC0A4F8
2026-08-03 20:50:53.948 UTC [234620] LOG:  database system is ready to accept connections

Three differences compared to the first test:

• The “last known up at” timestamp (20:44:36) does not match a checkpoint we ran manually. It matches the last automatic checkpoint triggered during the load.

• The redo is not instantaneous anymore: 3.63 seconds, replaying about 280 MB of WAL (from LSN 20/7A3618B0 to 20/8BC0A4D0). All the write activity between the last checkpoint and the snapshot had to be replayed. The transactions committed before the snapshot are recovered, the ones that were in flight are rolled back.

• The end-of-recovery checkpoint then writes everything the redo rebuilt in memory: 105355 buffers, 53.6% of the buffer pool. The database is ready to accept connections about 19 seconds after the service start.

The crash recovery completed correctly and the database has been restored. We verify the TPC-B invariant. Each pgbench transaction applies the same delta to the accounts, tellers and branches tables in a single transaction. On a consistent database, the three sums must be equal:

Major drawbacks

• The snapshot covers the whole zvol. All the databases of the cluster are captured and restored together. There is no per-database restore, unlike the METADATA_ONLY approach on SQL Server which targets a single database.
• There is no backup history. SQL Server records the metadata backup in msdb. Here, the only trace is the snapshot itself on the ZFS side.
• Point-in-time recovery is not covered. The snapshot alone brings the database back to the moment it was taken. For PITR, WAL archiving would still be required on top of it.

Conclusion

• The snapshot backup and restore model of the SQL Server series applies to PostgreSQL (no I/O freeze, no metadata backup).
• One important rule: data files and WAL must reside on the same zvol so the snapshot is atomic.
• A 146 GB database was restored in a few seconds and in less than 20 seconds under active load, WAL replay included.

Thank you. Amine Haloui

L’article PostgreSQL Snapshot Backup and Restore with Proxmox ZFS (4/4) est apparu en premier sur dbi Blog.

In this guide, I walk through a real-world in-place upgrade using Leapp, covering preparation, resolving high-severity warnings, executing the upgrade, recompiling PostgreSQL, fixing collation mismatches, and restoring Patroni.

I. Preparation

Before the upgrade, you must ensure the current OS is healthy and fully patched.

1. Pause High Availability

Prevent Patroni from triggering a failover during the reboot cycles. If you are using a single PostgreSQL cluster, stop it by stopping the service or by using pg_ctl stop.

patronictl -c /etc/patroni/patroni.yml pause
systemctl stop patroni

2. Fix Subscription & Perform Full Update

I’m using an old VM for this blog, and If just like me, you see 403 Forbidden errors on repositories like codeready-builder, refresh your registration:

[root@patroni2 ~]# sudo dnf update -y
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs)                                                                                                     761  B/s | 480  B     00:00
Errors during downloading metadata for repository 'codeready-builder-for-rhel-9-x86_64-rpms':
  - Status code: 403 for https://cdn.redhat.com/content/dist/rhel9/9/x86_64/codeready-builder/os/repodata/repomd.xml (IP: 23.206.57.92)
Error: Failed to download metadata for repo 'codeready-builder-for-rhel-9-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

[root@patroni2 ~]# sudo subscription-manager clean
[root@patroni2 ~]# sudo subscription-manager register --force
[root@patroni2 ~]# sudo subscription-manager attach --auto
[root@patroni2 ~]# sudo subscription-manager refresh

[root@patroni2 ~]# sudo dnf update -y
...
Complete!

[root@patroni2 ~]# reboot

II. The Leapp Upgrade to 10.1

1. Install & Analyze

In this first phase, we are preparing the system for a major in-place upgrade using Leapp, the official upgrade framework for Red Hat–based distributions. When we install the package:

[root@patroni2 ~]# dnf install leapp-upgrade -y
...
Installed:
  leapp-0.20.0-1.el9.noarch                leapp-deps-0.20.0-1.el9.noarch          leapp-upgrade-el9toel10-0.23.0-1.el9.noarch       leapp-upgrade-el9toel10-deps-0.23.0-1.el9.noarch
  libdb-utils-5.3.28-57.el9_6.x86_64       python3-leapp-0.20.0-1.el9.noarch       systemd-container-252-55.el9_7.7.x86_64

Complete!

When running leapp preupgrade –target 10.1, we are not performing the upgrade. Instead, Leapp performs a full system audit to determine if the server is ready for RHEL 10.1. It checks:

• Installed packages and their compatibility
• Deprecated or removed libraries
• Kernel drivers that will not exist in RHEL 10
• Bootloader configuration (GRUB2)
• GPG key validity
• Custom system-level modifications (like dynamic linker changes)
• …

Think of this step as a dry-run with intelligence.

[root@patroni2 ~]# sudo leapp preupgrade --target 10.1

...

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. GRUB2 core will be automatically updated during the upgrade
    2. Detected customized configuration for dynamic linker.
    3. Leapp detected loaded kernel drivers which are no longer maintained in RHEL 10.
    4. Failed to read GPG keys from provided key files
    5. Berkeley DB (libdb) has been detected on your system

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       4
    MEDIUM severity reports:     1
    LOW severity reports:        1
    INFO severity reports:       3

Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
    A report has been generated at /var/log/leapp/leapp-report.txt
    A report has been generated at /var/log/leapp/leapp-report.json

After running the pre-upgrade analysis, the next step is to carefully review:

/var/log/leapp/leapp-report.txt

What we are looking for first is simple:

• Errors: 0
• Inhibitors: 0

If an Inhibitor is present, the upgrade will be blocked entirely.
In my case, there were no blockers, but I did have several high severity warnings.

High severity does not mean the upgrade will fail.
It means: This could break something, review it carefully.

Let’s look at one concrete example from my system.

2. High Severity Example – Dynamic Linker Customization

Leapp detected that my system had a custom dynamic linker configuration:

Risk Factor: high

Title: Detected customized configuration for dynamic linker.

Summary: Custom configurations to the dynamic linker could potentially impact the upgrade in a negative way. The custom configuration includes modifications to /etc/ld.so.conf, custom or modified drop in config files in the /etc/ld.so.conf.d directory and additional entries in the LD_LIBRARY_PATH or LD_PRELOAD variables. These modifications configure the dynamic linker to use different libraries that might not be provided by Red Hat products or might not be present during the whole upgrade process. The following custom configurations were detected by leapp:

- The following drop in config files were marked as custom:

    - /etc/ld.so.conf.d/postgres.conf

Remediation: [hint] Remove or revert the custom dynamic linker configurations and apply the changes using the ldconfig command. In case of possible active software collections we suggest disabling them persistently.

Key: cc9bd972af70b7a27f66a37b11a00dcfcb73b1bc

----------------------------------------

What does this actually mean?

The dynamic linker (ld.so) is responsible for loading shared libraries at runtime.

By modifying:

• /etc/ld.so.conf
• files in /etc/ld.so.conf.d/
• LD_LIBRARY_PATH
• LD_PRELOAD

we are telling the system to load non-standard or custom libraries. In PostgreSQL environments (especially with custom builds or extensions), this is common practice. However, during a major OS upgrade, these custom paths might:

• Point to libraries that do not exist in RHEL 10
• Override new system libraries
• Break dependency resolution mid-upgrade

Leapp flags this because it cannot guarantee consistency during the transition phase. In my case, it shouldn’t be an issue, because inside postgres.conf, I only have a path aiming to the lib directories of my PostgreSQL installation, which will not change, but we will still see how to prevent an error.

Understanding the Remediation

The report clearly suggests:

Remove or revert the custom dynamic linker configurations and apply the changes using the ldconfig command.

In my case, the configuration was related to PostgreSQL, so temporarily removing it is safe for the upgrade preparation phase. Instead of deleting it permanently, I moved it aside:

[root@patroni2 ~]# sudo mv /etc/ld.so.conf.d/postgres.conf /tmp/postgres.conf.bak
[root@patroni2 ~]# sudo ldconfig

ldconfig rebuilds the system library cache now based only on standard paths.

Re-Run the Preupgrade Check

After remediation, always re-run the preupgrade command and check the report again. If the fix was successful:

• The severity should disappear from the REPORT OVERVIEW
• The issue should no longer appear in leapp-report.txt

This validation loop is important. We are progressively cleaning the system until it is fully compliant for upgrade.

============================================================
                      REPORT OVERVIEW
============================================================

HIGH and MEDIUM severity reports:
    1. Leapp detected loaded kernel drivers which are no longer maintained in RHEL 10.
    2. GRUB2 core will be automatically updated during the upgrade
    3. Failed to read GPG keys from provided key files
    4. Berkeley DB (libdb) has been detected on your system

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       3
    MEDIUM severity reports:     1
    LOW severity reports:        1
    INFO severity reports:       3

Before continuing, review the full report below for details about discovered problems and possible remediation instructions:
    A report has been generated at /var/log/leapp/leapp-report.txt
    A report has been generated at /var/log/leapp/leapp-report.json

Only once the report is clean, or fully understood, should we proceed to the actual upgrade execution.

2. Execute the upgrade

Once all errors and inhibitors are resolved, and high-severity findings have been reviewed or remediated, we are finally ready to perform the actual in-place upgrade. This is the moment where Leapp transitions from analysis mode to execution mode.

About the Repository Warning

During the preupgrade phase, Leapp informed us that codeready-builder-… repositories are not officially supported during the upgrade process and are excluded by default.

This is expected behavior as Leapp only enables a minimal, controlled set of repositories to ensure:

• Package consistency
• Dependency resolution stability
• Predictable upgrade paths

However, in PostgreSQL environments, some packages (extensions, development headers, libraries) may depend on CodeReady Builder. If a repository is truly required during the upgrade, we must explicitly enable it using:

--enablerepo <repoid>

Running the Upgrade

Since I need CodeReady Builder for PostgreSQLdependencies, I ran:

leapp upgrade --target 10.1 --enablerepo codeready-builder-for-rhel-10-x86_64-rpms

What happens when we run this command?

At this stage, Leapp:

• Resolves and downloads required RHEL 10 packages
• Builds a temporary upgrade environment
• Prepares a special upgrade initramfs
• Modifies the bootloader (GRUB) to boot into the upgrade environment on next reboot

The system is not upgraded immediately. The actual OS transition happens during the next boot.

After the command completes, Leapp generates another report. Just like in the preupgrade phase, verify:

• Errors: 0
• Inhibitors: 0

If everything looks clean, we can proceed.

Reboot – The Real Upgrade Begins

# reboot

This is where the real upgrade starts. During boot:

• The system enters a temporary upgrade environment
• Packages are replaced
• Obsolete components are removed
• Configuration files are migrated
• The new RHEL 10 kernel is installed

This phase can take several minutes depending on your VM/server resources. My VM doesn’t have many resources and it took me around 30 minutes. Be patient, interrupting this process can leave the system in an inconsistent state.

Verifying the Upgrade

Once the server is back online, confirm the OS version:

[root@patroni2 ~]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="10.1 (Coughlan)"
ID="rhel"
ID_LIKE="centos fedora"
VERSION_ID="10.1"
PLATFORM_ID="platform:el10"
PRETTY_NAME="Red Hat Enterprise Linux 10.1 (Coughlan)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:10.1"
HOME_URL="https://www.redhat.com/"
VENDOR_NAME="Red Hat"
VENDOR_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT=”Red Hat Enterprise Linux 10″
REDHAT_BUGZILLA_PRODUCT_VERSION=10.1
REDHAT_SUPPORT_PRODUCT=”Red Hat Enterprise Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=”10.1″

This confirms that we are now running RHEL 10.1.

III. Post-Upgrade Database Recovery

Once you login to RHEL 10.1, your Postgres binaries in /u01 will fail because libicuuc.so.67 (from RHEL 9) is missing. It can also fail because of other libraries.

14:45:55 postgres@patroni2:/home/postgres/ [test-op-patroni] pgstart
/u01/app/postgres/product/17/db_6/bin/postgres: error while loading shared libraries: libicuuc.so.67: cannot open shared object file: No such file or directory
no data was returned by command ""/u01/app/postgres/product/17/db_6/bin/postgres" -V"
command not found
program "postgres" is needed by pg_ctl but was not found in the same directory as "/u01/app/postgres/product/17/db_6/bin/pg_ctl"

1. Recompile PostgreSQL

Since you installed from source, you must re compile PostgreSQL with the new RHEL 10 system libraries. Here is the command I personally use to build it, with the postgres user:

postgres@patroni2:/home/postgres/ [dummy] MAJOR="17"
postgres@patroni2:/home/postgres/ [dummy] MINOR="6"
postgres@patroni2:/home/postgres/ [dummy] tar axf postgresql-${MAJOR}.${MINOR}.tar.gz
postgres@patroni2:/home/postgres/ [dummy] mkdir build; cd $_
postgres@patroni2:/home/postgres/ [dummy] export PGHOME="/u01/app/postgres/product/${MAJOR}/db_${MINOR}"
postgres@patroni2:/home/postgres/ [dummy] export SEGSIZE=2
postgres@patroni2:/home/postgres/ [dummy] export BLOCKSIZE=8
postgres@patroni2:/home/postgres/ [dummy] meson setup . ../postgresql-${MAJOR}.${MINOR}
postgres@patroni2:/home/postgres/ [dummy] meson configure -Dprefix=${PGHOME}                   -Dbindir=${PGHOME}/bin                   -Ddatadir=${PGHOME}/share                   -Dincludedir=${PGHOME}/include                   -Dlibdir=${PGHOME}/lib                   -Dsysconfdir=${PGHOME}/etc                   -Dpgport=5432                   -Dplperl=enabled                   -Dplpython=enabled                   -Dssl=openssl                   -Dpam=enabled                   -Dldap=enabled                   -Dlibxml=enabled                   -Dlibxslt=enabled                   -Dsegsize=${SEGSIZE}                   -Dblocksize=${BLOCKSIZE}                   -Dllvm=enabled                   -Duuid=ossp                   -Dzstd=enabled                   -Dlz4=enabled                   -Dzstd=enabled                   -Dgssapi=enabled                   -Dsystemd=enabled                   -Dicu=enabled                   -Dsystem_tzdata=/usr/share/zoneinfo                   -Dextra_version=" dbi services build"
postgres@patroni2:/home/postgres/ [dummy] ninja
postgres@patroni2:/home/postgres/ [dummy] ninja install

2. Restore Library Paths

[root@patroni2 ~]# mv /tmp/postgres.conf.bak /etc/ld.so.conf.d/postgres.conf
[root@patroni2 ~]# ldconfig

3. Start & Fix Collation Mismatch

Postgres will now start, but will warn you about Collation Version Mismatches (2.34 vs 2.39).

15:09:55 postgres@patroni2:/home/postgres/build/ [test-op-patroni] pgstart
waiting for server to start.... done
server started
15:10:28 postgres@patroni2:/home/postgres/build/ [test-op-patroni] psql
WARNING:  database "postgres" has a collation version mismatch
DETAIL:  The database was created using collation version 2.34, but the operating system provides version 2.39.
HINT:  Rebuild all objects in this database that use the default collation and run ALTER DATABASE postgres REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
psql (17.6 dbi services build)
Type "help" for help.

Inside Postgres, run for every database:

ALTER DATABASE postgres REFRESH COLLATION VERSION;
ALTER DATABASE

-- Repeat for other DBs if applicable
REINDEX DATABASE postgres;

Your PostgreSQL is now starting properly and your server has been upgraded.

4. In case of a patroni cluster

Since the system Python version has changed after the OS upgrade, your old .local venv is invalid. You must recreate it. Here is how I install patroni using the postgres user:

$ python3 -m venv .local
$ .local/bin/pip3 install –upgrade pip
$ .local/bin/pip3 install –upgrade setuptools
$ .local/bin/pip3 install wheel
$ .local/bin/pip3 install psycopg[binary]
$ .local/bin/pip3 install python-etcd
$ .local/bin/pip3 install patroni
$ .local/bin/patroni version

Check if the cluster sees the member again. If patronictl list is empty, a restart of the service is usually required to re-register with etcd.

10:45:40 postgres@patroni2:/home/postgres/ [test-op-patroni] patronictl list
+ Cluster: test-op-patroni (7565882985963789761) -+-----+------------+-----+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------+------+------+-------+----+-------------+-----+------------+-----+
+--------+------+------+-------+----+-------------+-----+------------+-----+
10:45:48 postgres@patroni2:/home/postgres/ [test-op-patroni] sudo systemctl restart patroni
10:45:55 postgres@patroni2:/home/postgres/ [test-op-patroni] patronictl list
+ Cluster: test-op-patroni (7565882985963789761) --+---------+----+-------------+-----+------------+-----+
| Member                 | Host           | Role   | State   | TL | Receive LSN | Lag | Replay LSN | Lag |
+------------------------+----------------+--------+---------+----+-------------+-----+------------+-----+
| patroni-tst-op-geapg02 | 192.168.56.142 | Leader | running | 11 |             |     |            |     |
+------------------------+----------------+--------+---------+----+-------------+-----+------------+-----+

IV. Conclusion

Upgrading from RHEL 9.6 to 10.1 is a big move. It’s not just a simple update; it’s a total shift in the system’s foundation. Between hardware driver changes and library updates, you really have to pay attention to the details to keep your database running.

RHEL 10.1 is a great, modern platform, but you can’t just click “update” and hope for the best. By planning the upgrade, planning to rebuild your binaries, refreshing your database objects, you can make the jump without the drama. Take the pre-upgrade report seriously, it’s there for a reason!

That said, for a production PostgreSQL cluster, especially one managed with Patroni and etcd, I would not recommend this in-place upgrade approach. Even if Leapp makes the process technically possible, you are still:

• Modifying the operating system in place
• Replacing core libraries underneath a running database stack
• Trusting automated dependency resolution during a major version jump

In production, risk reduction should always be the priority.

Instead, I strongly recommend provisioning new VMs or physical servers, installing RHEL 10.1 from scratch, deploying PostgreSQL, Patroni, and etcd cleanly, rebuilding the cluster from best practices, and then migrating the data from the old environment to the new one using replication or another appropriate method.

Sometimes the safest upgrade… is a new cluster.

L’article Upgrade RHEL from 9.6 to 10.1 (when running PostgreSQL/Patroni) est apparu en premier sur dbi Blog.

Monitoring a PostgreSQL environment involves much more than simply checking whether the database is running. In a modern high availability architecture, multiple components work together to keep the service available and stable.

In this article, we will explain how to monitor a PostgreSQL cluster environment with Zabbix:

• PostgreSQL
• Patroni
• HAProxy
• ETCD
• Keepalived

The goal of this blog is to provide a simple and clear overview that can be understood by both beginners and experienced administrators.

PostgreSQL High Availability Architecture

A typical PostgreSQL high availability environment often contains several components:

In a PostgreSQL HA environment, monitoring only the database is not enough because every component directly impacts cluster stability and application availability. Moreover, a failures in components like HAProxy, ETCD, or Keepalived can directly impact application availability and cluster stability. Comprehensive monitoring ensures the entire ecosystem remains healthy and operational.

Monitoring PostgreSQL with Zabbix

Zabbix already provides an official PostgreSQL template.

This template allows administrators to monitor:

• Database availability
• Connections
• Transactions
• Cache hit ratio
• Replication status
• Locks
• Deadlocks
• Database size
• Query statistics
• WAL activity
• Performance metrics

Therefore, using the official template is usually the fastest and easiest way to start monitoring PostgreSQL.

ETCD

Zabbix already provides an official template for ETCD monitoring.

The template uses HTTP Agent items to collect metrics directly from the /metrics endpoint exposed by ETCD.

In addition, the template works without external scripts and uses Prometheus-style metrics collection.

The official template already includes many useful triggers and metrics for monitoring ETCD health, performance, and cluster activity.

HAProxy

HAProxy plays a critical role in a PostgreSQL high availability environment, ensuring that client connections are automatically routed to the correct PostgreSQL node and maintaining application connectivity during failovers.

Zabbix already provides an official HAProxy monitoring template that can be used as a base for monitoring and customization: HAProxy by HTTP

The number of active servers in HAProxy is also monitored by the template. However, the template does not create a trigger by default for this metric. This means that a backend could lose one or more active servers without generating any alert in Zabbix. For this reason, it is highly recommended to create custom triggers on this item to immediately detect backend availability issues and avoid unnoticed service degradation.

To implement this properly, a new macro should be created in the template to define the minimum expected number of active servers. Then, a custom trigger must be added on the item prototype available in the Low-Level Discovery (LLD) section of the template. Consequently, Zabbix can automatically monitor and alert on all discovered HAProxy backends.

Patroni

Zabbix does not directly include Patroni monitoring in the standard PostgreSQL templates.

One approach is to use the Zabbix Systemd template, which allows monitoring of services running on the server. In this case, it is simply necessary to filter on the patroni service name to verify that the service is active.

Another approach is to create a custom item in a template using a Zabbix agent item with the following key:

net.tcp.service["{$HAPROXY.STATS.SCHEME}","{$HAPROXY.STATS.HOST}","{$HAPROXY.STATS.PORT.PATRONI}"]

This item performs a direct HTTP TCP connection test on port 5000, which is commonly used by the Patroni API. This configuration allows Zabbix to verify that the Patroni service responds correctly and remains reachable.

Additionally, combining both monitoring methods significantly improves monitoring reliability.

Monitoring Keepalived

Keepalived monitoring is not included by default in the standard Zabbix templates.
To properly monitor the VIP management layer, administrators should create custom items whether the Virtual IP (VIP) is correctly present on one of the cluster nodes.

A common approach is to create three different items:

• Two items using the net.tcp.service key to test TCP connectivity for:
  • the READ-ONLY access
  • the READ-WRITE access
• One additional item using a system.run command to directly verify the VIP status on the server.

Together, these checks help ensure that:

• the VIP is correctly assigned
• the PostgreSQL services are reachable
• the failover mechanism is working properly

Finally, administrators should create the appropriate triggers for these items in order to immediately detect VIP failover or connectivity issues.

Conclusion

In conclusion, monitoring PostgreSQL alone is not enough in a high availability environment.

A complete monitoring strategy must include:

• PostgreSQL
• Patroni
• HAProxy
• ETCD
• Keepalived

Using Zabbix with both official and custom templates provides a centralized and efficient monitoring solution.

The PostgreSQL template available in Zabbix already offers excellent database monitoring capabilities.

Additional templates for ETCD and HAProxy help extend monitoring to the full HA architecture and improve visibility across the entire platform.

As a result, this approach helps administrators detect failures earlier, improve troubleshooting, and ensure better service availability.

If you need you can try other blogs regarding Zabbix or postgreSQL here: dbi Blog

L’article Zabbix – Monitoring full cluster Patroni/PostgreSQL est apparu en premier sur dbi Blog.

And yet, somewhere downstream, your change-data-capture pipeline has just gone quiet. Debezium or Flink CDC was streaming every change out of that database into Kafka, into a data lake, into a search index. After the failover it stops. No crash, no alert, just a lag counter that climbs and a topic that never gets new records. By the time someone notices, you are missing hours of changes and the only honest fix is a full reseed.

This is one of my favourite operational traps, because it sits in the blind spot between two teams that both think they did their job. The DBA built a solid HA cluster. The data engineer built a solid CDC pipeline. Nobody owned the join between them, and the join is a logical replication slot.

From a DBA’s perspective this is a silent killer, and PostgreSQL 17 finally gave us a clean way to disarm it. Let me show you exactly why it breaks, what the old workarounds cost, and how native failover slots fix it, with a small Patroni lab you can run on Docker Desktop to see both the failure and the cure.

If you only run Debezium: your half is one line, slot.failover=true on the connector (Debezium ≥ 3.0.5 against a PostgreSQL 17+ primary). The other half is server-side, two settings and the failover wiring, and it is your DBA’s job; the rest of this post is what to ask them for.

Why the slot does not survive

A logical replication slot is a bookmark. It records how far a consumer has read the write-ahead log (the WAL is PostgreSQL’s running log of every change; an LSN is just a position in it, like a byte offset), and it keeps PostgreSQL from recycling WAL the consumer still needs. Debezium, a native CREATE SUBSCRIPTION, pg_recvlogical all rely on a named slot to guarantee they never miss a change and never have to replay from the beginning.

The issue is that a logical slot lives on only one instance, the one where it was created. Physical streaming replication, the mechanism Patroni uses to keep a standby in sync, copies the data blocks and the WAL. It does not copy the logical slots. For years that was a deliberate design decision, not a bug: a slot carries decoding state and a catalog_xmin (a do-not-garbage-collect marker for the old system-catalog rows it still needs to decode the WAL), and replaying that safely onto a standby is genuinely hard.

So the picture before failover looks healthy:

The slot exists only on the primary. Promote the standby, and the new primary has all your data and none of your slots. Debezium reconnects through your load balancer, asks for cdc_slot, and PostgreSQL answers, correctly, that no such slot exists. The pipeline is broken, and depending on how the connector is configured it either fails loudly or, worse, quietly recreates a fresh slot at the current position and skips everything in between.

Can’t my load balancer or VIP just handle this?

This is the first question I get, and it is a good one. The instinct is reasonable : I already have HAProxy in front of the cluster, or a VIP that floats with the leader on keepalived, or a cloud load balancer, so when the primary moves my consumer simply reconnects to the new one. Doesn’t that cover it?

It does not, and understanding why tells you what class of problem this really is.

A load balancer solves routing: how the consumer finds whoever is currently the primary. A floating VIP does the same thing one layer down. Both get the client to the right door, and you do need one of them (in the lab that is exactly HAProxy’s job). But the slot is not a routing concern, it is state. It is a file in the primary’s data directory. When the consumer reconnects through the VIP to the new primary and asks for cdc_slot, that instance still has to physically own the slot. The IP moving does not move the slot, because the new primary is a different PostgreSQL instance with its own pg_replslot directory. So whether HAProxy sits on the database nodes or on its own tier, and whether you use HAProxy or a VIP or a cloud LB, the effect on the slot is identical: none.

There is exactly one architecture where the slot survives without any slot synchronization at all, and it is a different design rather than a different load balancer: shared-storage failover. If your HA is built on a block volume that detaches from the failed node and reattaches to the new one (DRBD, a SAN, a cloud disk, a Kubernetes PVC), then failover moves the whole data directory, pg_replslot included, and the slot comes along because it never logically moved. That works, but it buys a different set of trade-offs (the storage becomes a hard dependency and a fencing problem, and you give up streaming read replicas off the same volume), and it is not what a streaming-replication cluster like Patroni does. On streaming replication the slot has to be synchronized deliberately, which is the rest of this post.

Why not just create the slot on every node?

The next idea people reach for is to pre-create the slot on every node, primary and standbys, and let the consumer use whichever one is currently primary. The intuition is right. In fact it is almost exactly what failover slots do, it just misses the one piece that makes it safe.

A slot is not only a name. It carries moving state: restart_lsn, confirmed_flush_lsn and catalog_xmin. If you create independent slots on each node, each one starts at that node’s position and then sits there, because nothing advances the standby’s copy as the consumer acknowledges changes on the primary. The names match, the positions drift.

That is worse than losing the slot, not better. After failover the consumer finds a slot called cdc_slot (so no loud error) and resumes from its stale position. If the copy is behind the consumer, you re-deliver changes it already processed, which is duplicates. If it is ahead, the consumer asks for an LSN the slot has already discarded, which is a gap or an error. You have turned a loud, obvious failure(“replication slot does not exist”, which at least pages someone) into a silent data-integrity problem, and for a pipeline feeding a data lake, silent is the worst outcome.

Two more walls. Before PostgreSQL 16 you cannot even create a logical slot on a standby (you get “logical decoding cannot be used while in recovery”); version 16 added logical decoding on standbys, and 17 added the synchronization that makes a standby slot actually track the primary. And idle slots are not free: every slot, active or not, pins WAL and holds back vacuum on its node, so leaving un-advanced slots on every node invites WAL growth.

So the correct version of “a slot on every node, one active and the rest idle” is precisely a failover slot: present on the standby, not consumable until promotion, with its position kept in lockstep with the primary. The synchronization is the whole feature.

The years before PostgreSQL 17

This problem is not new, and the community built two workarounds.

The first was the pg_failover_slots extension (from EDB), which runs on theprimary and copies logical slot positions out to physical standbys. It supportsPostgreSQL 11 and up, but from 17 on you would reach for the native feature instead,which is what EDB now recommends too. Either way it is an external extension you haveto install, load viashared_preload_libraries, and keep compatible with your major version.

The second, if you run Patroni, is built in. Patroni has its own concept of permanent replication slots. You declare a slot in the cluster configurationand Patroni makes sure it exists on every eligible node, copying the slot to thestandbys and advancing its position on a schedule. In Patroni terms it looks like this:

slots:
  cdc_slot:
    type: logical
    database: cdcdemo
    plugin: pgoutput

This genuinely survives a switchover, and for a lot of clusters it has been good enough for years. But notice the words “on a schedule”. Patroni advances the copy of the slot on the standby every loop_wait seconds, by default every 10 seconds. That periodic copy is the weak point. The standby’s idea of “how far the consumer has read” can lag the real primary by up to a loop interval, so a failover at the wrong moment can leave the new primary slightly behind what the consumer already acknowledged, which is exactly the window where you get duplicates or, with the wrong settings, a gap. It is a copy advanced on a timer, not a hard guarantee, so itis stale between refreshes and nothing stops the consumer from running ahead of it.

That is the gap PostgreSQL 17 closes.

PostgreSQL 17 failover slots

PostgreSQL 17 introduced native failover slots: logical slots that the database itself keeps synchronized to a physical standby as part of replication itself, with a guarantee that the consumer cannot read past what the standby has (the synchronized_standby_slots part below), rather than an external copy on a timer. (The feature landed in 17, so everything here applies to PostgreSQL 17 and later. I built the lab on PostgreSQL18, which adds a couple of conveniences I will mention at the end.)

There are three moving parts.

1. Mark the slot as a failover slot. pg_create_logical_replication_slot grew a fifth argument,failover:

SELECT * FROM pg_create_logical_replication_slot('cdc_slot', 'pgoutput',
                                                  false, false, true);
--                                       temporary --^     ^      ^
--                                          twophase ------+      |
--                                          failover -------------+

If you drive logical replication with a subscription, the same flag exists there, CREATE SUBSCRIPTION ... WITH (failover = true), and the slot and the subscription must agree. If you use Debezium 3.x, you set slot.failover = true and the connector creates the slot with the flag for you, as long as it is talking to a PostgreSQL 17+ primary.

2. Let the standby synchronize slots. On the standby you enable the slot sync worker (both are replication settings):

sync_replication_slots = on
hot_standby_feedback   = on

hot_standby_feedback is not optional here. The standby has to tell the primary which catalog rows it still needs so the primary does not vacuum away rows the synchronized slot depends on.

3. Hold the logical slot back until the standby has the data. On the primary you set synchronized_standby_slots:

synchronized_standby_slots = 'physical_slot_of_the_standby'

This is the subtle, important one, a leash on the consumer. It tells the primary’s logical WAL senders never to hand a change to the CDC consumer until the physical standby has also received it, so the consumer can never get ahead of the standby. Without it, after failover the new primary cannot give the consumer what it already saw; with it, promotion is always safe.

With those in place, the slot is mirrored in real time. On the standby, pg_replication_slots shows it as synced, and it cannot be consumed until the standby is promoted:

 slot_name | slot_type | active | failover | synced | confirmed_flush_lsn
-----------+-----------+--------+----------+--------+---------------------
 cdc_slot  | logical   | f      | t        | t      | 0/3A1240

failover = t, synced = t, active = f. That row is the whole point. When this standby becomes primary, cdc_slot is already there, at the right LSN, and the consumer reconnects as if nothing happened.

The gotcha that bites everyone, silently

There is a prerequisite that is easy to miss and fails in the quietest possible way: the standby’s primary_conninfo must contain a dbname (the logical replication failover docs call this out, but it is easy to skip past).

A physical replication connection does not need a database name, so historically nobody put one there. But the slot sync worker connects to the primary as a normal backend to read slot state, and it needs a database to connect to. If dbname is missing, the worker does not error in your face. It just does not sync anything. Your slots silently fail to appear on the standby, and you find out at failover, which is the worst possible time.

Recent Patroni adds dbname to primary_conninfo for you when it is managing logical slots, but I check it explicitly, every time, and so should you, on the standby:

SHOW primary_conninfo;  

It must contain a dbname=… ; if not, slot sync is silently doing nothing

Seeing it break, then survive

Let me walk both cases on a small two-node Patroni cluster: PostgreSQL 18 nodespatroni1 and patroni2, a Debezium connector reading through HAProxy (so it always follows the current primary), and a tiny generator inserting rows with a strictly increasing counter n. The proof is simple: compare the n values that reached Kafka against the rows actually in the table. Every committed row should be in Kafka; duplicates are fine, because CDC delivery is at-least-once.

The commands below are the real thing, run them on your own cluster if you like. If you would rather watch it happen in a few minutes, there is a ready-to-run Docker version at the end of this post.

First, the break: a plain slot

A normal logical slot has failover off, the default. Debezium creates one for you when you register a connector with slot.failover=false; the bare SQL equivalent is:

SELECT pg_create_logical_replication_slot('dbz_slot', 'pgoutput');

It lives on the primary and nowhere else, because nothing mirrors it:

-- on the primary (patroni1)
SELECT slot_name, slot_type, active, failover, synced FROM pg_replication_slots;

 slot_name | slot_type | active | failover | synced
-----------+-----------+--------+----------+--------
 dbz_slot  | logical   | t      | f        | f
 patroni2  | physical  | t      | f        | f        (the standby's streaming slot)

On the standby, dbz_slot is simply not there. Now fail over with patronictl:

patronictl -c /etc/patroni/patroni.yml switchover --leader patroni1 --candidate patroni2 --force cdc-demo

Successfully switched over to "patroni2"

The new primary never had the slot. The consumer reconnects through HAProxy and, here is the trap, stays perfectly green:

curl -s localhost:8083/connectors/orders-plain/status | jq '{state: .connector.state, task: .tasks[0].state}'

{ "state": "RUNNING", "task": "RUNNING" }

because Debezium found its slot missing and silently recreated it at the current LSN, straight from the Connect log:

Creating replication slot with command CREATE_REPLICATION_SLOT "dbz_slot" LOGICAL pgoutput

Everything committed between the switchover and that recreation is skipped. Draining the topic and comparing it to the source table shows the hole:

table rows: 255    distinct rows in Kafka: 250    missing: 5   (n = 100..104)

Five committed rows, gone, and nothing anywhere raised an error. Whether Debezium recreates the slot quietly (as here) or fails loudly is a matter of configuration; the lab shows the quiet case because it is the one that hurts. That is the silent version that costs you a reseed three days later when someone notices the warehouse is short.

Then, the cure: a failover slot

Same scenario, two changes. First, create the slot with failover on (or set Debezium’s slot.failover=true, which does it for you):

SELECT pg_create_logical_replication_slot('cdc_slot', 'pgoutput', false, false, true);

Second, turn on synchronization. On the standby:

sync_replication_slots = on
hot_standby_feedback   = on

and on the primary, hold logical slots back until the standby has the data, naming the standby’s physical slot:

ALTER SYSTEM SET synchronized_standby_slots = 'patroni2';
SELECT pg_reload_conf();

Now the slot is mirrored to the standby. Look at it there:

-- on the standby (patroni2)
SELECT slot_name, active, failover, synced, temporary, confirmed_flush_lsn
  FROM pg_replication_slots WHERE slot_name = 'cdc_slot';

 slot_name | active | failover | synced | temporary | confirmed_flush_lsn
-----------+--------+----------+--------+-----------+---------------------
 cdc_slot  | t      | t        | t      | f         | 0/34B0A10

failover = t, synced = t. One gotcha worth watching: a freshly synced slot starts temporary = t and is dropped on promotion until it persists. The standby log shows the transition, a transient refusal (its catalog_xmin is briefly ahead of the primary slot’s) and then sync-ready:

LOG:  could not synchronize replication slot "cdc_slot"
DETAIL:  ... the remote slot needs catalog xmin 760, but the standby has xmin 761.
LOG:  newly created replication slot "cdc_slot" is sync-ready now

Only once temporary = f is it safe to fail over. So do it:

patronictl -c /etc/patroni/patroni.yml switchover --leader patroni1 --candidate patroni2 --force cdc-demo

On the new primary the slot is already there and live, and the consumer resumes on its own. Re-point synchronized_standby_slots at the new standby (the correct value flips after a role change), then reconcile:

-- on the new primary
ALTER SYSTEM SET synchronized_standby_slots = 'patroni1';
SELECT pg_reload_conf();

table rows: 616    distinct rows in Kafka: 616    missing: 0   (74 duplicates)

Every committed row reached Kafka. Same cluster, same switchover, same consumer; the only difference is one boolean on the slot and the synchronization around it. The duplicates are at-least-once redelivery (you may see a handful or none on a given run), so the consumer must be idempotent either way, but not one committed row was lost. That boolean is the gap between “we lost four hours of CDC” and “nobody noticed”.

Patroni does its part, but not all of it

I want to be honest about the rough edges, because this is where the real-world write-ups tend to go quiet.

PostgreSQL 17 gives you the mechanism. Patroni, as of the 4.1 series I tested, does not yet fully wire it for you, and there are two manual pieces.

First, Patroni does not manage synchronized_standby_slots (this is tracked as Patroni issue #3431). You have to set it yourself, on the primary, to the physical slot name of the current standby. And in a two-node cluster that value is not symmetric: when patroni1 leads it must name patroni2‘s slot, and after a switchover to patroni2 it must name patroni1‘s slot. So you cannot just bake a single static value into the cluster config, you have to reset it after a role change. The lab’s enable-failover-config.sh is role-aware and does this, and the walkthrough deliberately runs it again after the switchover so you can see the value flip. In production you would automate it with a Patroni callback on role change.

Second, the dbname in primary_conninfo requirement I mentioned. Recent Patroni handles it, but it is exactly the kind of thing that is true on the version you tested and silently false on the version you deployed, so verify it rather than trust it.

Neither of these is a reason to avoid failover slots. They are a reason to treat “failover slots on Patroni” as a thing you configure and test deliberately, not a checkbox you assume.

The checklist

If you run logical replication or CDC on a Patroni cluster, here is what I would verify :

Where this is going: PG18 today, PG19 next

The lab runs on PostgreSQL 18, which keeps the PG17 mechanism intact and adds a few conveniences around it: pg_recvlogical --enable-failover lets you create a failover slot straight from the command-line consumer, and --enable-two-phase (plus pg_createsubscriber --enable-two-phase) rounds out the tooling. The configuration is identical to 17, so everything above applies unchanged.

PostgreSQL 19 is in development as I write this, so treat this as a direction, not a promise (check the final release notes). The theme is telling: most of the 19 cycle’s work here makes the failure I hit easier to see rather than changing the mechanism, slot-sync skip counters in pg_stat_replication_slots, a pg_sync_replication_slots()that no longer fails silently, starting logical replication without a wal_level restart (a new effective_wal_level), and logical replication of sequences.

What 19 does not appear to change is the manual piece I complained about. synchronized_standby_slots still has to be set, and reset after a role change, by something outside PostgreSQL, and the Patroni side of that (issue #3431) is still open. So the checklist above does not get shorter in 19, it gets easier to debug.

Try it yourself

Everything above is in a lab attached to this post, worklab-patroni-failover-slots: a Docker Compose stack (etcd, two PostgreSQL 18 Patroni nodes, HAProxy, Kafka in KRaft mode, and a Debezium connector) plus a handful of helper scripts that wrap exactly the commands shown here.

We spend a lot of effort making the database survive a node failure, and then we quietly assume everything hanging off the database survives with it. Logical replication slots are the reminder that high availability is not only about the primary coming back, it is about everything that depends on the primary coming back too.

PostgreSQL 17 failover slots close the last real gap between PostgreSQL HA and the CDC pipelines we increasingly build on top of it. So don’t sleep too much on PG14 version (EOL November this year !). The mechanism is solid, the configuration is small, and the failure it prevents is the kind that is invisible until it is expensive. Clone the lab, break it once with a plain slot, fix it once with a failover slot, and you will never look at a Patroni switchover the same way again. When I started by DBA journey I have always been told that if you want to know how an RDBMS works playing around backups and restores would provide you deep insights into how they are running. I feel like today this is more true for replication.

Link for the scripts.

L’article Surviving a Patroni failover with logical replication est apparu en premier sur dbi Blog.

In Part 1 of this series, we covered what pgvector is, how embeddings work, and how to store them in PostgreSQL. We ended with a working similarity search — but on a sequential scan. That works fine for a demo table with 1,000 rows. It does not work for production.

This post is about what comes next: indexes. Specifically, the three index families in the pgvector ecosystem as of February 2026 (HNSW, IVFFlat, and DiskANN), including two DiskANN implementations targeting different deployment models, what they’re good at, where they break, and the patterns you need, whether you’re the DBA tuning them or the developer looking to understand the the strenghts of PostgreSQL as a vector store.

Everything in this post was tested on public dataset: 25,000 Wikipedia articles embedded with OpenAI’s text-embedding-3-large at 3,072 dimensions, the maximum the model supports. The high number of dimension is a choice, to highlight some limitations for pedagogical reasons. You would be ok running and testing with lower dimensions or other embedding models, you might want to look into the RAG series, I will probably make a blog post on how to test embedding models against your data sets.
The environment is PostgreSQL 18 with pgvector 0.8.1 and pgvectorscale 0.9.0.

All the SQL scripts, Python code, and Docker configuration are in the companion lab: lab/06_pgvector_indexes.

The Index Types

Before we dive in, here’s the landscape. pgvector ships with two built-in index types (HNSW and IVFFlat), and two DiskANN implementations are available from different vendors:

That pgvectorscale number is not a typo. 21 MB vs 193 MB for the same data. W

Note: This post uses pgvectorscale for all DiskANN benchmarks since it’s the open-source, self-hosted option. We’ll compare both DiskANN implementations in detail in Section 3. pg_diskann is available only for Azure Flexible Server for PostgreSQL, the managed instance service from Microsoft.

HNSW: The Default Choice

HNSW (Hierarchical Navigable Small World) is the most commonly recommended index type for vector search. It builds a multi-layered graph where each node connects to its nearest neighbors, and search navigates this graph from top to bottom.

The 2,000-Dimension Wall

Here’s the first thing you’ll hit with modern embedding models:

CREATE INDEX idx_content_hnsw
ON articles USING hnsw (content_vector vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

ERROR:  column cannot have more than 2000 dimensions for hnsw index

The vector type in pgvector has a 2,000-dimension limit for HNSW indexes. If you’re using text-embedding-3-large (3,072 dimensions), or voyage-3-large at its 2,048-dimension setting, this is a blocker.

The workaround: halfvec.

-- Step 1: Store a half-precision copy
ALTER TABLE articles ADD COLUMN content_halfvec halfvec(3072);
UPDATE articles SET content_halfvec = content_vector::halfvec;

-- Step 2: Index the halfvec column (limit: 4,000 dimensions)
CREATE INDEX idx_content_hnsw_halfvec
ON articles USING hnsw (content_halfvec halfvec_cosine_ops)
WITH (m = 16, ef_construction = 64);

Time: 28974.392 ms (00:28.974)

SELECT pg_size_pretty(pg_relation_size('idx_content_hnsw_halfvec')) AS hnsw_size;

 hnsw_size
-----------
 193 MB

29 seconds to build, 193 MB for 25,000 articles at 3,072 dimensions. That’s roughly 8 KB per row in the index alone.

Important: halfvec stores each dimension in 2 bytes instead of 4. You lose some floating-point precision, but in practice the recall difference is negligible for similarity search. The storage savings are real: your halfvec column is 6 KB per row vs 12 KB for the full vector.

Alternative: Instead of a separate column, you can create an expression index that casts on the fly: CREATE INDEX ... ON articles USING hnsw ((content_vector::halfvec(3072)) halfvec_cosine_ops); The trade-off is that your queries must use the matching expression (content_vector::halfvec(3072) <=> ...) for the planner to pick it up, which is harder to read in application code. The separate column approach gives cleaner queries.

Tuning ef_search: The Recall vs Speed Dial

ef_search controls how many candidates the HNSW algorithm considers during search. Higher values mean more candidates examined, better recall, but more work. The default is 40.

-- ef_search = 40 (default)
SET hnsw.ef_search = 40;
EXPLAIN ANALYZE
SELECT id, title, content_halfvec <=> (
    SELECT content_halfvec FROM articles WHERE id = 1
) AS distance
FROM articles
ORDER BY content_halfvec <=> (
    SELECT content_halfvec FROM articles WHERE id = 1
)
LIMIT 10;

Index Scan using idx_content_hnsw_halfvec on articles
  Order By: (content_halfvec <=> (InitPlan 1).col1)
  Index Searches: 1
  Buffers: shared hit=551
Execution Time: 6.004 ms

-- ef_search = 100
SET hnsw.ef_search = 100;

Index Scan using idx_content_hnsw_halfvec on articles
  Buffers: shared hit=716
Execution Time: 2.365 ms

-- ef_search = 200
SET hnsw.ef_search = 200;

Index Scan using idx_content_hnsw_halfvec on articles
  Buffers: shared hit=883
Execution Time: 2.542 ms

Wait — ef_search=100 was faster than ef_search=40? Not really. Those numbers came from a warm cache (shared hit=551, zero disk reads). The apparent speedup is a cache warming effect, not a property of the algorithm. To prove this, I restarted PostgreSQL and ran the full sweep from cold:

The second run at ef_search=40 clocked 0.8 ms — faster than any ef_search=100 or 200 run. On a warm cache, all three (40/100/200) land in the 0.8-5 ms range. The variance is cache state, not algorithmic shortcuts. The real cost cliff is at ef_search=400 where the optimizer switches plans entirely:

-- ef_search = 400
SET hnsw.ef_search = 400;

Sort  (actual time=364.693..364.695 rows=10.00 loops=1)
  Sort Key: ((articles.content_halfvec <=> (InitPlan 1).col1))
  Sort Method: top-N heapsort  Memory: 25kB
  ->  Seq Scan on articles  (actual time=0.143..356.482 rows=24700.00 loops=1)
        Filter: (content_halfvec IS NOT NULL)
        Rows Removed by Filter: 300
Execution Time: 364.724 ms

The planner chose a Seq Scan + Sort path. At ef_search=400, PostgreSQL’s cost model estimated the index scan would be more expensive than reading every row sequentially, and the query went from 2.5 ms to 365 ms.

This is your optimizer flip-flop. It’s not a bug — it’s the planner doing its job. But it means you need to be aware of the threshold.

DBA Takeaway: For HNSW on halfvec(3072), stay in the ef_search 40-200 range. Past that, you’re fighting the optimizer. If you need ef_search > 200 for recall, you probably need a bigger m parameter at build time.

Build Parameters: m and ef_construction

m is the number of connections per node in the graph. ef_construction is the candidate list size during build. Higher values = better graph quality but slower builds and (potentially) larger indexes.

-- m=16, ef_construction=64 (default-ish)
Time: 28,974 ms    Size: 193 MB

-- m=32, ef_construction=128
Time: 54,077 ms    Size: 193 MB

At 25K rows, doubling m doubled the build time but didn’t change the index size. The size effect becomes more visible at larger scales. In general: start with m=16, ef_construction=64 and only increase if recall is insufficient after tuning ef_search.

IVFFlat: Fast Build

IVFFlat (Inverted File with Flat quantization) partitions the vector space into Voronoi cells using k-means clustering, then searches only the cells closest to the query vector.

Same Dimension Limit

IVFFlat has the same 2,000-dimension limit as HNSW for the vector type. Same workaround:

CREATE INDEX idx_content_ivfflat
ON articles USING ivfflat (content_halfvec halfvec_cosine_ops)
WITH (lists = 25);

Time: 5008.765 ms (00:05.009)

SELECT pg_size_pretty(pg_relation_size('idx_content_ivfflat'));
-- 193 MB

5 seconds to build vs 29 for HNSW. The index size is nearly identical (193 MB vs 193 MB), but IVFFlat builds 5.8x faster.

The lists parameter controls the number of Voronoi cells. The pgvector documentation recommends: rows / 1000 for tables up to 1M rows, sqrt(rows) for larger tables. For 25,000 articles: 25000 / 1000 = 25 lists, giving roughly 1,000 rows per cell. A common mistake is applying sqrt(rows) to small tables — that gives 158 lists here, creating cells with only ~167 rows each, which fragments the index and causes the optimizer to flip to sequential scan at surprisingly low probe counts.

Tuning Probes: How Many Cells to Search

probes controls how many Voronoi cells are searched at query time. Default is 1 — fast but low recall. A good starting point is sqrt(lists). Here’s the sweep with lists=25:

SET ivfflat.probes = 1;
-- Index Scan, Execution Time: 1.0 ms, Buffers: 571

SET ivfflat.probes = 2;
-- Index Scan, Execution Time: 3.7 ms, Buffers: 1,944

SET ivfflat.probes = 3;
-- Index Scan, Execution Time: 4.4 ms, Buffers: 2,793

SET ivfflat.probes = 4;
-- Index Scan, Execution Time: 5.9 ms, Buffers: 3,937

And then:

SET ivfflat.probes = 5;

Sort  (actual time=152.548..152.549 rows=10.00 loops=1)
  Sort Key: ((articles.content_halfvec <=> (InitPlan 1).col1))
  ->  Seq Scan on articles  (actual time=0.144..148.844 rows=24700.00 loops=1)
        Filter: (content_halfvec IS NOT NULL)
Execution Time: 152.584 ms

Same story as HNSW. At probes=5 (searching 5 of 25 cells = 20%), the optimizer decided a sequential scan was cheaper. The query went from 6 ms to 153 ms.

Takeaway: For IVFFlat with 25 lists, the optimizer flips between probes=4 and probes=5. With sqrt(25) = 5, you’re right at the tipping point. Use SET LOCAL ivfflat.probes = 3 or 4 for a good recall/speed balance. On larger tables (100K+ rows), the flip happens at much higher probe counts because sequential scans become proportionally more expensive.

SET LOCAL: The Production Pattern

Never set ivfflat.probes at the session level in production. Use SET LOCAL inside a transaction:

BEGIN;
    SET LOCAL ivfflat.probes = 3;
    SELECT id, title, content_halfvec <=> (
        SELECT content_halfvec FROM articles WHERE id = 1
    ) AS distance
    FROM articles
    ORDER BY content_halfvec <=> (
        SELECT content_halfvec FROM articles WHERE id = 1
    )
    LIMIT 10;
COMMIT;

-- Verify: probes reverted to default
SHOW ivfflat.probes;  -- 1

The setting reverts automatically after COMMIT/ROLLBACK. No global state leakage. Do the same for hnsw.ef_search.

DO NOT PLAY AROUND WITH SESSION PARAMETERS ON PRODUCTION. Use SET LOCAL inside a transaction, or set them at the function/procedure level. Session-level changes persist until disconnect and can affect every query on that connection, including connection pooler reuse. Hinting is not a strategy.

DiskANN: Leveraging B-TREE index principle

DiskANN is provided by the pgvectorscale project from Timescale (SQL extension name: vectorscale). It implements the DiskANN algorithm with Statistical Binary Quantization (SBQ) compression built in.

No 2,000-Dimension Wall

Unlike HNSW and IVFFlat, DiskANN supports the vector type natively up to 16,000 dimensions — no halfvec workaround needed:

CREATE INDEX idx_content_diskann ON articles USING diskann (content_vector)
WITH (storage_layout = memory_optimized);

NOTICE:  Starting index build with num_neighbors=50, search_list_size=100,
         max_alpha=1.2, storage_layout=SbqCompression.  -- memory_optimized maps to SBQ
NOTICE:  Indexed 24700 tuples
Time: 49140.736 ms (00:49.141)

SELECT pg_size_pretty(pg_relation_size('idx_content_diskann'));
-- 21 MB

49 seconds to build, but 21 MB. Compare:

        indexname         |  size  | size_bytes
--------------------------+--------+------------
 idx_content_diskann      | 21 MB  |   22,511,616
 idx_content_hnsw_halfvec | 193 MB |  202,350,592
 idx_content_ivfflat      | 193 MB |  202,522,624

DiskANN is 9x smaller than HNSW and IVFFlat on the same data. SBQ compression is the default — storage_layout = memory_optimized is what you get if you don’t specify a layout. Specifying it explicitly (as in the CREATE INDEX above) is good practice for readability. The alternative plain layout stores full vectors in the index and does not compress.

Query performance is competitive:

Index Scan using idx_content_diskann on articles
  Order By: (content_vector <=> (InitPlan 1).col1)
  Buffers: shared hit=1437 read=132
Execution Time: 2.915 ms

3 ms for a 3,072-dimension nearest neighbor search on 25K articles. On the same data, HNSW does it in 2-6 ms and IVFFlat in 2-10 ms. All three are in the same ballpark for query speed.

Takeaway: DiskANN is the right choice when your HNSW index outgrows shared_buffers. At 193 MB for 25K rows, HNSW on halfvec(3072) would reach ~77 GB at 10 million rows, this is well beyond what most buffer pools can keep hot. DiskANN’s 9x compression keeps the navigational structure cached while full vectors stay in the heap, fetched only for the final rescore of a few dozen candidates. Same access pattern as a B-tree: compact index in memory, selective heap lookups on demand. The trade-off is longer build times and fewer operator class options (vector type only, no halfvec/bit/sparsevec).

CREATE INDEX CONCURRENTLY

As of pgvectorscale 0.9.0, DiskANN supports CONCURRENTLY:

CREATE INDEX CONCURRENTLY idx_content_diskann
ON articles USING diskann (content_vector)
WITH (storage_layout = memory_optimized);

This is critical for production — you can build the index without locking the table. HNSW and IVFFlat also support CREATE INDEX CONCURRENTLY.

Why These Dimension Limits Exist: Buffer Pages and Bits

If the 2,000 / 4,000 / 16,000 dimension limits seem arbitrary, they’re not. The intuition comes from how PostgreSQL stores data: in 8 KB buffer pages (8,192 bytes). Every index tuple — including the vector representation in a vector index — has to fit within a page. The fewer bytes per dimension, the more dimensions you can pack.

Here’s the back-of-the-envelope arithmetic:

The theoretical numbers explain the intuition — why halfvec doubles the limit and quantized encodings push it further. The actual limits are slightly lower because of page headers, tuple overhead, and the index metadata stored alongside each vector. For HNSW, each page also stores neighbor connection lists (up to m * 2 neighbor IDs per node); for IVFFlat, each page carries centroid references and list pointers. These eat into the available space, which is why pgvector’s HNSW sets 2,000 (not 2,048) and pgvectorscale’s DiskANN sets 16,000 (not 16,384). But the pattern is unmistakable: the fewer bits per dimension, the more dimensions you can fit in a page.

This is why DiskANN can handle 16,000 dimensions where HNSW on vector tops out at 2,000 — DiskANN stores a compressed representation in the index page, not the full vector. And why halfvec doubled the HNSW/IVFFlat limit from 2,000 to 4,000: half the bytes per dimension, twice the capacity.
This is more than enough for the vast majority of use cases. Most embedding models in production today default to 768–1536 dimensions, well within the 2,000-dimension limit. This also proves how future proof the curent vector store implementation is on PostgreSQL.

How pgvectorscale Compresses: Statistical Binary Quantization (SBQ)

pgvectorscale’s DiskANN from Tiger Data uses a method called Statistical Binary Quantization (SBQ). The idea is deceptively simple: for each dimension, replace the float value with a 1 or 2-bit code.

1-bit mode (default for dimensions >= 900): Each dimension is compressed to a single bit. But not naively — standard binary quantization uses 0.0 as the threshold (positive = 1, negative = 0), which works poorly because real embedding distributions are rarely centered on zero. SBQ instead computes the per-dimension mean across all vectors during index build and uses that as the threshold:

if value > mean_of_this_dimension → 1
else → 0

A 3,072-dimension float32 vector (12,288 bytes) becomes a 3,072-bit string (384 bytes). That’s 32x compression. During search, the query vector is also SBQ-encoded, and distances are computed using XOR + popcount on the bit strings — which modern CPUs execute in a single instruction.

2-bit mode (default for dimensions < 900): Each dimension gets two bits, encoding four “zones” based on the z-score (how many standard deviations from the mean). This gives finer granularity at 16x compression instead of 32x.

The accuracy loss is real but small. On common benchmarks, SBQ achieves 96-99% recall compared to exact search. The rescore step (controlled by diskann.query_rescore, default 50) compensates: after the graph traversal finds the top-50 candidates using quantized distances, pgvectorscale fetches the full-precision vectors from the heap and re-computes exact distances to produce the final top-10.

What’s stored where:

This two-tier architecture is why DiskANN achieves 21 MB index size: the index only stores 384-byte compressed vectors, not 12 KB originals. The full vectors stay in the table, fetched only for the final re-ranking of a few dozen candidates.

Microsoft’s pg_diskann: A Different Approach

There’s a second DiskANN implementation for PostgreSQL: Microsoft’s pg_diskann, currently documented and distributed for Azure Database for PostgreSQL Flexible Server. It uses the same Vamana graph algorithm but a fundamentally different compression strategy: Product Quantization (PQ).

Where SBQ asks “is this dimension above or below the mean?”, Product Quantization asks “which of 16 codewords best represents this group of dimensions?”

Here’s how PQ works, step by step:

1. Divide the vector into chunks. A 3,072-dimension vector is split into, say, 1,024 chunks of 3 dimensions each.
2. Train a codebook per chunk. For each chunk, k-means clustering finds 16 representative codewords (centroids). Why 16? Because 16 values fit in 4 bits — a single hex digit.
3. Encode each chunk as a 4-bit symbol. During index build, each chunk of 3 dimensions is replaced by the index (0-15) of its closest codeword. The entire 3,072-dimension vector becomes 1,024 symbols of 4 bits each = 512 bytes.
4. Decode via lookup table at query time. To compute the distance to a query vector, you pre-compute the distance from the query to all 16 codewords for each chunk, creating a 16-row x 1,024-column lookup table. Then for each stored vector, you sum up the table entries corresponding to its symbols. No floating-point multiplication needed — just table lookups and additions.

The compression is dramatic: 3,072 dimensions * 4 bytes = 12,288 bytes → 512 bytes with PQ. That’s 24x compression, in the same ballpark as SBQ’s 32x.

Comparing the Two DiskANN Implementations

Both implementations share the core DiskANN algorithm (Vamana graph) and the two-phase search pattern (compressed scan + full-precision rescore). The difference is how they compress:

• SBQ is simpler and faster to build (just compute means). It’s a blunt instrument — 1 bit per dimension loses a lot of information, but XOR + popcount is blazingly fast, and the rescore step recovers accuracy.
• PQ is more sophisticated and retains more information per bit (a 4-bit symbol captures relationships between groups of dimensions). It’s slower to build (k-means training) but can achieve better recall at the same compression ratio, especially for vectors with correlated dimensions.

Takeaway: If you’re self-hosting PostgreSQL, pgvectorscale is your DiskANN option — open source, well-maintained, and the SBQ compression is effective. If you’re on Azure Database for PostgreSQL, you also have pg_diskann, whose PQ compression may give better recall on very high-dimensional data. The underlying algorithm is the same; the compression strategy is the difference.

Iterative Scans

This is the most important query-time feature added to pgvector since HNSW support. If you take one thing from this post, let it be this section.

The Problem

Vector search indexes return the K nearest neighbors, then PostgreSQL applies your WHERE clause. If your filter is selective, you get fewer results than requested.

-- Our dataset has 9 categories with varying selectivity
  category   | cnt   | % of 25K
-------------+-------+---------
 History     | 8719  | 34.9%
 General     | 6221  | 24.9%
 Science     | 2232  |  8.9%
 Mathematics |  116  |  0.5%

When you search for the 10 nearest “Science” articles, the HNSW index returns its ef_search nearest neighbors (say, 40), PostgreSQL filters to keep only Science articles, and you get however many matched. For Science (8.9%), you’ll usually get your 10 results. For Mathematics (0.5%), you won’t.

Let’s prove it. Here’s a search for Mathematics articles with ef_search=40, forcing the HNSW index path. (We disable sequential scan here to force the index path on our small 25K dataset. On production tables with 100K+ rows, the optimizer would naturally choose the index without this hint.)

SET enable_seqscan = off;
SET hnsw.iterative_scan = 'off';
SET hnsw.ef_search = 40;

SELECT id, title, category,
       content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1) AS distance
FROM articles
WHERE category = 'Mathematics'
ORDER BY content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1)
LIMIT 10;

Index Scan using idx_content_hnsw_halfvec on articles
  Order By: (content_halfvec <=> (InitPlan 1).col1)
  Filter: (category = 'Mathematics'::text)
  Rows Removed by Filter: 40
  Index Searches: 1
Execution Time: 0.766 ms
(0 rows)

Zero rows returned. The index fetched 40 candidates, all 40 were filtered out (none were Mathematics), and the query silently returned an empty result set. This is the core problem: your application asked for 10 results and got 0.

Before iterative scans, you had two bad options:

1. Over-fetch (LIMIT 1000) and hope enough rows match — wasteful and unreliable
2. Sequential scan — correct but slow on large tables

The Solution: Iterative Index Scans

pgvector 0.8.0 introduced iterative scans. Instead of fetching a fixed batch and filtering, the index keeps fetching more candidates until the filter is satisfied.

SET enable_seqscan = off;
SET hnsw.iterative_scan = 'relaxed_order';
SET hnsw.ef_search = 40;

SELECT id, title, category,
       content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1) AS distance
FROM articles
WHERE category = 'Mathematics'
ORDER BY content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1)
LIMIT 10;

Index Scan using idx_content_hnsw_halfvec on articles
  Order By: (content_halfvec <=> (InitPlan 1).col1)
  Filter: (category = 'Mathematics'::text)
  Rows Removed by Filter: 6809
  Index Searches: 171
Execution Time: 4235.805 ms
(10 rows)

10 rows returned. The index scanned 171 times, examined 6,819 candidates, filtered out 6,809 non-Mathematics rows, and delivered exactly 10 results. It took 4.2 seconds — much slower than the 0.8 ms empty result — but you got a correct answer instead of silence.

At 6,819 tuples scanned, this was well within the default max_scan_tuples of 20,000. On a table with millions of rows and the same 0.5% selectivity, the scan might hit the 20,000 limit before finding 10 matches — you’d get a partial result set. That’s the trade-off the safety valve makes: bounded latency vs guaranteed result count.

That 4.2-second cost reflects the extreme selectivity: Mathematics is only 0.5% of the data, so the index had to traverse deep into the graph. For moderate selectivity like Science (8.9%), the overhead is negligible:

SET hnsw.iterative_scan = 'relaxed_order';
SET hnsw.ef_search = 40;

SELECT ... WHERE category = 'Science' ORDER BY ... LIMIT 10;

Index Scan using idx_content_hnsw_halfvec on articles
  Order By: (content_halfvec <=> (InitPlan 1).col1)
  Filter: (category = 'Science'::text)
  Rows Removed by Filter: 26
  Index Searches: 1
Execution Time: 1.058 ms

Same feature, but for Science the index found 10 matching rows in a single search pass — no extra work needed.

Two Modes

• relaxed_order: Results are approximately ordered by distance. Slightly faster. Good enough for most use cases.
• strict_order: Results are exactly ordered by distance. Slightly slower. Use when ranking precision matters. SET hnsw.iterative_scan = ‘strict_order’;
  — Execution Time: 0.885 ms

The Safety Valve: max_scan_tuples

To prevent runaway scans on extremely selective filters (imagine filtering for a category that has 1 row in 10 million), there’s a safety limit:

SET hnsw.max_scan_tuples = 500;   -- Restrictive: stop after 500 index tuples
SET hnsw.max_scan_tuples = 20000; -- Default: generous enough for most workloads
SET hnsw.max_scan_tuples = 0;     -- Unlimited (use with caution)

IVFFlat Too

Same concept, different GUC prefix:

SET ivfflat.iterative_scan = 'relaxed_order';
SET ivfflat.probes = 3;

SELECT ... WHERE category = 'Science' ORDER BY ... LIMIT 10;
-- Execution Time: 2.001 ms

Filtered Results in Action

With iterative scan, every result matches the filter:

SET hnsw.iterative_scan = 'relaxed_order';
SET hnsw.ef_search = 100;

SELECT id, title, category,
       round((content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1))::numeric, 4) AS distance
FROM articles
WHERE category = 'Science'
ORDER BY content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1)
LIMIT 10;

  id   |  title   | category | distance
-------+----------+----------+----------
     1 | April    | Science  |   0.0000
  7862 | April 23 | Science  |   0.2953
  9878 | April 25 | Science  |   0.3076
   469 | May      | Science  |   0.3082
  9880 | April 24 | Science  |   0.3451
   402 | July     | Science  |   0.3453
  5156 | April 4  | Science  |   0.3531
  9530 | April 7  | Science  |   0.3588
 34906 | 2013     | Science  |   0.3674
  9149 | April 8  | Science  |   0.3690

All 10 results are Science. All sorted by cosine distance. No over-fetching, no sequential scan. (The titles are Wikipedia date articles — “April”, “May”, etc. — that happen to be classified under Science in this dataset.)

Multi-Filter Combinations

Iterative scans work with compound WHERE clauses too:

SET hnsw.iterative_scan = 'relaxed_order';
SET hnsw.ef_search = 100;

SELECT id, title, category, word_count
FROM articles
WHERE category = 'Science' AND word_count > 1000
ORDER BY content_halfvec <=> (SELECT content_halfvec FROM articles WHERE id = 1)
LIMIT 10;

Limit  (actual time=7.098..7.105 rows=10.00 loops=1)
  ->  Sort  (actual time=7.097..7.101 rows=10.00 loops=1)
        Sort Key: ((content_halfvec <=> (InitPlan 1).col1))
        Sort Method: top-N heapsort  Memory: 25kB
        ->  Bitmap Heap Scan on articles  (actual time=0.690..7.027 rows=444.00 loops=1)
              Recheck Cond: ((word_count > 1000) AND (category = 'Science'::text))
              ->  BitmapAnd  (actual time=0.641..0.643 rows=0.00 loops=1)
                    ->  Bitmap Index Scan on idx_articles_word_count  (rows=1806.00)
                    ->  Bitmap Index Scan on idx_articles_category    (rows=2232.00)
Execution Time: 7.542 ms

Here the PostgreSQL optimizer did something clever: it combined two B-tree indexes (BitmapAnd) instead of using the HNSW iterative scan. On a 25K-row table that fits in memory, this is often cheaper. At scale (millions of rows), the iterative scan path wins because the bitmap approach requires reading too many heap pages.

Takeaway: Iterative scans are the answer to “vector search + WHERE doesn’t work.” Enable them with SET LOCAL hnsw.iterative_scan = 'relaxed_order' inside transactions. The safety valve max_scan_tuples = 20000 is a sensible default.

Quantization and Storage

With 3,072-dimension embeddings, storage is the elephant in the room. Here’s what each representation costs per row:

SELECT
  pg_size_pretty(avg(pg_column_size(content_vector)))  AS vector_size,
  pg_size_pretty(avg(pg_column_size(content_halfvec))) AS halfvec_size,
  pg_size_pretty(avg(pg_column_size(content_bq)))      AS binary_size
FROM articles WHERE content_vector IS NOT NULL;

 vector_size | halfvec_size | binary_size
-------------+--------------+-------------
 12 kB       | 6148 bytes   | 392 bytes

At 1 million rows:

The storage math is brutal for high-dimensional embeddings. This is why DiskANN’s built-in SBQ compression matters: it gets you to 21 MB where HNSW on halfvec costs 193 MB.

Binary Quantize + Re-Ranking

Binary quantization crushes each dimension to a single bit (positive = 1, negative = 0). It’s lossy, but very fast for coarse filtering. The pattern is a two-phase search:

-- Phase 1: Hamming distance on binary (fast, coarse) → 100 candidates
-- Phase 2: Cosine distance on full vector (precise) → 10 results

WITH coarse AS (
    SELECT id, title, content_vector
    FROM articles
    WHERE content_bq IS NOT NULL
    ORDER BY content_bq <~> (
        SELECT binary_quantize(content_vector)::bit(3072)
        FROM articles WHERE id = 1
    )
    LIMIT 100
)
SELECT id, title,
       content_vector <=> (SELECT content_vector FROM articles WHERE id = 1) AS distance
FROM coarse
ORDER BY content_vector <=> (SELECT content_vector FROM articles WHERE id = 1)
LIMIT 10;

Limit  (actual time=29.021..29.026 rows=10.00 loops=1)
  ->  Sort on coarse  (actual time=29.020..29.023 rows=10.00 loops=1)
        ->  Subquery Scan  (actual time=27.805..28.999 rows=100.00 loops=1)
              ->  Sort by content_bq <~>  (actual time=27.744..27.754 rows=100.00 loops=1)
                    ->  Seq Scan on articles  (actual time=0.435..24.288 rows=24700.00 loops=1)
Execution Time: 29.127 ms

29 ms without an index on content_bq. With an HNSW index using bit_hamming_ops, Phase 1 would be sub-millisecond. The re-ranking in Phase 2 only touches 100 full vectors instead of 25,000.

DBA Takeaway: Use halfvec as your default for 3072-dimension embeddings. Use binary quantize + re-ranking when you need to search billions of rows and can tolerate a two-phase approach. DiskANN’s SBQ gives you similar compression automatically.

Operators and Sargability

pgvector provides four distance operators:

Wrong Operator = No Index

This is the single most common mistake. If your index uses halfvec_cosine_ops but your query uses <-> (L2), the index cannot be used:

-- CORRECT: cosine operator on cosine index → Index Scan
EXPLAIN (COSTS OFF)
SELECT ... ORDER BY content_halfvec <=> (...) LIMIT 10;

 Limit
   ->  Index Scan using idx_content_hnsw_halfvec on articles
         Order By: (content_halfvec <=> (InitPlan 1).col1)

-- WRONG: L2 operator on cosine index → Seq Scan!
EXPLAIN (COSTS OFF)
SELECT ... ORDER BY content_halfvec <-> (...) LIMIT 10;

 Limit
   ->  Sort
         Sort Key: ((articles.content_halfvec <-> (InitPlan 1).col1))
         ->  Seq Scan on articles

The planner can’t use a cosine-distance index for an L2-distance query. They’re different metrics with different orderings. If you see an unexpected Seq Scan on a vector query, check your operator first.

Sargable Queries: The Cross-Join Trap

This is the pattern I see most often in the wild, and it’s wrong:

-- BAD: cross-join prevents index use
SELECT a.id, a.title,
       a.content_halfvec <=> b.content_halfvec AS distance
FROM articles a, articles b
WHERE b.id = 1
ORDER BY a.content_halfvec <=> b.content_halfvec
LIMIT 10;

Limit
  ->  Sort
        Sort Key: ((a.content_halfvec <=> b.content_halfvec))
        ->  Nested Loop
              ->  Index Scan using articles_pkey on articles b
                    Index Cond: (id = 1)
              ->  Seq Scan on articles a     ← NO INDEX!

The planner sees a.content_halfvec <=> b.content_halfvec as a join condition, not an index-scan ordering. It can’t push the ORDER BY into the vector index because the right-hand side comes from a different table reference.

The fix: use a scalar subquery:

-- GOOD: scalar subquery → index used
SELECT id, title,
       content_halfvec <=> (
           SELECT content_halfvec FROM articles WHERE id = 1
       ) AS distance
FROM articles
ORDER BY content_halfvec <=> (
    SELECT content_halfvec FROM articles WHERE id = 1
)
LIMIT 10;

Limit
  InitPlan 1
    ->  Index Scan using articles_pkey on articles articles_1
          Index Cond: (id = 1)
  ->  Index Scan using idx_content_hnsw_halfvec on articles
        Order By: (content_halfvec <=> (InitPlan 1).col1)

The scalar subquery is evaluated once (InitPlan), then the result is treated as a constant. Now the planner can push the ORDER BY into the HNSW index scan. Same results, but with the index instead of a sequential scan.

Takeaway: For this nearest-neighbor ORDER BY pattern, always use scalar subqueries for vector distance calculations, never cross-joins. This is the vector-search equivalent of writing sargable WHERE clauses for B-tree indexes.

Partial Indexes

If you frequently filter by a specific category, a partial index is dramatically more efficient:

CREATE INDEX idx_content_hnsw_science
ON articles USING hnsw (content_halfvec halfvec_cosine_ops)
WITH (m = 16, ef_construction = 64)
WHERE category = 'Science';

Time: 1420.207 ms (00:01.420)

        indexname         |  size
--------------------------+--------
 idx_content_hnsw_halfvec | 193 MB   ← full table (25,000 rows)
 idx_content_hnsw_science |  17 MB   ← Science only (2,232 rows)

11x smaller, 20x faster to build. If your application queries consistently filter by a known set of categories, partial indexes are the single biggest optimization available. Build one per high-value category.

Monitoring

Index Inventory

SELECT indexname,
       pg_size_pretty(pg_relation_size(indexname::regclass)) AS size,
       indexdef
FROM pg_indexes
WHERE tablename = 'articles'
ORDER BY pg_relation_size(indexname::regclass) DESC;

        indexname        |  size   | indexdef
-------------------------+---------+-------------------------------------------
 idx_content_hnsw_halfvec| 193 MB  | USING hnsw ... WITH (m='16', ef_construction='64')
 idx_content_ivfflat     | 193 MB  | USING ivfflat ... WITH (lists='25')
 idx_content_diskann     | 21 MB   | USING diskann ... WITH (storage_layout=memory_optimized)
 articles_pkey           | 1384 kB | USING btree (id)
 idx_articles_category   | 1192 kB | USING btree (category)
 idx_articles_word_count | 904 kB  | USING btree (word_count)

Adding up the three vector indexes (193 + 193 + 21 = 407 MB) plus B-tree indexes (3 MB), the total index footprint is over 410 MB for a 90 MB table. The indexes are ~4.5x the data. This is typical for high-dimensional vector data — plan your storage accordingly.

Note: In practice you’d pick one vector index, not all three. With just HNSW + B-tree indexes, the ratio drops to ~2x.

Settings

Know what settings exist and what they default to:

SELECT name, setting, short_desc
FROM pg_settings
WHERE name ~ '^(hnsw|ivfflat|diskann)\.'
ORDER BY name;

                    name                    | setting | short_desc
--------------------------------------------+---------+-------------------------------------------
 hnsw.ef_search                             | 40      | Dynamic candidate list size for search
 hnsw.iterative_scan                        | off     | Mode for iterative scans
 hnsw.max_scan_tuples                       | 20000   | Max tuples to visit for iterative scans
 hnsw.scan_mem_multiplier                   | 1       | Multiple of work_mem for iterative scans
 ivfflat.iterative_scan                     | off     | Mode for iterative scans
 ivfflat.max_probes                         | 32768   | Max probes for iterative scans
 ivfflat.probes                             | 1       | Number of probes
 diskann.query_search_list_size             | 100     | Search list size for queries
 diskann.query_rescore                      | 50      | Rescore candidates

Takeaway: hnsw.iterative_scan and ivfflat.iterative_scan default to off. If your application relies on vector search with WHERE clauses, you need to explicitly enable iterative scans.

Access Method Capabilities

Not sure which operator class works with which index type? Query the catalog:

SELECT am.amname AS access_method,
       opc.opcname AS operator_class,
       t.typname AS data_type
FROM pg_opclass opc
JOIN pg_am am ON am.oid = opc.opcmethod
JOIN pg_type t ON t.oid = opc.opcintype
WHERE am.amname IN ('hnsw', 'ivfflat', 'diskann')
ORDER BY am.amname, opc.opcname;

HNSW is the most versatile. DiskANN is the most constrained. IVFFlat falls in between.

Build Progress Monitoring

While building a large index:

SELECT phase,
       round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS pct_done,
       tuples_done, tuples_total
FROM pg_stat_progress_create_index;

This works for HNSW and IVFFlat builds (pgvector reports progress). DiskANN builds from pgvectorscale don’t currently report to this view.

Decision Guidelines

Here’s how to choose:

Start with HNSW unless you have a reason not to. It has the broadest operator support, good recall, and predictable performance. Use halfvec if your dimensions exceed 2,000.

Choose IVFFlat if build time matters more than query time. IVFFlat builds 5-6x faster than HNSW. If your data distribution shifts materially, plan a REINDEX to refresh clustering quality (centroid drift). A practical signal: if recall drops without any change in query patterns, or if you’ve inserted more than ~30% new rows since the last build, the centroids are likely stale. For a deeper look at how embedding model upgrades trigger reindexing at scale, see Embedding Versioning with pgvector.

Choose DiskANN if storage is the constraint. The 9x compression is decisive at scale. It handles high dimensions natively (no halfvec needed) and supports CONCURRENTLY for production deployments.

Enable iterative scans for vector search + filtering in production. The trade-off is potentially higher latency on very selective filters (the index does more work to find matching rows), but that’s usually the right trade-off for correctness. Tune max_scan_tuples / max_probes to bound worst-case work. Use relaxed_order by default, strict_order when ranking precision matters.

Use partial indexes for category-specific searches. An 11x size reduction and 20x faster build is hard to argue with.

Use SET LOCAL for all vector parameter changes in production AFTER having tested them.

Summary

Dimension limits are largely explained by PostgreSQL’s 8 KB page size and encoding density (exact cutoffs are implementation-defined):

The compression story is ultimately a story about how many bits of information you need per dimension to navigate the index. pgvector stores full-precision values; DiskANN stores just enough to find the right neighborhood, then goes back to the heap for exact distances.
But here’s the thing: exact results are not necessarily what you want. In a RAG pipeline, the retrieved documents are context for a language model that will synthesize and rephrase an answer — not return rows verbatim. Whether your top-10 results are ranked by exact cosine distance or by a 96%-accurate approximation rarely changes the generated answer. The same is true for recommendations, semantic deduplication, and most classification workflows: the downstream consumer is tolerant of small ranking variations.

The real production trade-off is not precision vs approximation — it’s the balance between retrieval speed, resource efficiency, and result quality at your scale. An HNSW index that doesn’t fit in shared_buffers and hits disk on every query will give you worse effective results than a DiskANN index that stays cached and returns slightly less precise distances in a fraction of the time. The best retrieval is the one that actually runs within your latency budget.

The lab with all SQL scripts, and Python embedding pipeline are available here : lab/06_pgvector_indexes.

All benchmarks: PostgreSQL 18, pgvector 0.8.1, pgvectorscale 0.9.0, 25K Wikipedia articles, 3072d text-embedding-3-large embeddings, (4 vCPUs, 8 GB RAM).

L’article pgvector, a guide for DBA – Part 2: Indexes (update march 2026) est apparu en premier sur dbi Blog.