A customer recently asked me to disable FileTable/FileStream on their instance because the databases no longer use it, and to ensure that this is indeed the case.
The aim is to migrate to a new architecture without FileTable/FileStream in a second time.

Link to FileTable and FileStream if you need to know a bit more about these technologies

The environment is in a high-availability (HA) configuration using AlwaysOn.

As this isn’t something we do every day, here is a 9-step procedure to achieve this:

•    Check that there are no more FileTables

•    Disable non-transactional access

•    Check that non-transactional access is disabled

•    Check the columns in the tables

•    Identify FILESTREAM filegroups & check FILESTREAM files

•    Delete FILESTREAM containers & the FILESTREAM filegroup

•    Disable FILESTREAM on the SQL Server instance

•    Disable FILESTREAM on the server

•    Disable FILESTREAM on the other servers

Let’s get started!

Step 1: Checking the FileTables

Let’s start with the first, fairly simple step, which is to check that there are no more FileTables

Query used:

SELECT name,object_id,is_filetable FROM <database>.sys.tables WHERE is_filetable = 1;

Example:

In this example, it is clear that there is no longer a FileTable in the tables of the two databases checked here.

Step 2: Disabling non-transactional access

To ensure that no further data is sent to FileTable/FileStream, we disable non-transactional access:

Query used:

ALTER DATABASE <database> SET FILESTREAM ( NON_TRANSACTED_ACCESS = OFF );

Example:

The next step is, of course, to check carefully that it is disabled.

Step 3: Check that non-transactional access is disabled

Query used:

SELECT db_name(database_id),* FROM sys.database_filestream_options;

Example:

As can be seen in the example, all databases are set to OFF

Step 4: Checking the columns in the tables

This is a very important step, as it is the one I missed in my example by the customer…

In fact, in step 6, if there is still a column in a table that has a FileStream defined, it is not possible to delete the associated files and filegroups.

Query used:

SELECT * FROM <database>.sys.columns WHERE is_filestream = 1;
select * from sys.objects where object_id= xxxx

The first query returns the columns that contain FileStream data, and the second provides information about the table.

You will then find a structure of the following type:
CREATE TABLE [schema].[table](
 [ID] [uniqueidentifier] ROWGUIDCOL  NOT NULL,
…..
[FS] [varbinary](max) FILESTREAM  NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] FILESTREAM_ON [FileStreamFileGroup]

We can therefore see that there is indeed a FileStream that needs to be removed at column level, but also at table level with the FILESTREAM_ON [FileStreamFileGroup].

Initially, creating a new column with [FS2] [varbinary](max) NULL, copying the data and renaming it, seemed like a good idea but was not sufficient, as the table still had FILESTREAM_ON [FileStreamFileGroup].
To resolve this, you need to recreate a table, copy the data and add the keys, indexes and constraints.

Then check everything again before proceeding to the next steps.

Example:

Step 5:  Identify FILESTREAM filegroups & Check FILESTREAM files

In this step, the aim is to identify the FILESTREAM filegroups defined in the databases so that they can be deleted.

Query used:

SELECT name, type_desc FROM <database>.sys.filegroups WHERE type = 'FD';
SELECT name,physical_name,type_desc FROM <database>.sys.database_files WHERE type_desc = 'FILESTREAM';

Example:

As there are two databases in my example that have a FileStream, there are four lines.

Step 6: Delete the FILESTREAM containers and the FILESTREAM FILEGROUP

Once identified, delete them using the following command.
Query used:

ALTER DATABASE [Database]  REMOVE FILE [FileStreamFile]

Example:

Step 7: Disable FILESTREAM on the SQL Server instance

At this point, we disable FILESTREAM at instance level.

Query used:

EXEC sp_configure 'filestream access level';
GO
EXEC sp_configure 'show advanced options', 1;
RECONFIGURE;
EXEC sp_configure 'filestream access level',0;
GO
RECONFIGURE;
GO
EXEC sp_configure 'show advanced options', 0;
RECONFIGURE;
GO

Example:

As can be seen from the example, the `config_value` is indeed 0, but the `run_value` has changed to 1.

In such cases, as always, we’ll check what the documentation says:

We are in T-SQL access only, it’s not really desactivated… Need a restart!

At this stage, we could also carry this out on the secondary node(s) of our AlwaysOn high availability cluster, but it is preferable to do it first on a single node and verify that it works.

Now it is time to disable Filestream on the SQL Server Engine service.

Step 8: Disabling FileStream on the server

For this step, we have already performed a failover of the AAGs to a synchronous secondary node.

We launch SQL Server Configuration Manager to disable FileStream in the Engine properties by unticking the box, as shown in the example above:

Then restart SQL Server and check that FileStream is disabled on the instance:

Step 9: Disabling FileStream on the other servers

At this point, you will just need to repeat steps 7 and 8 on the other nodes in your AlwaysOn high-availability cluster.

Conclusion

The steps are not complex in themselves, but where you can lose time is in the step 4 if you have to recreate a table without the FileStream and copy all data from this table. I was happy to have only one with not so much rows…

All the steps up to the restart of the SQL Server Engine service in step 8 are carried out online without any service interruption or failover.

I hope these steps will help anyone facing a similar issue!

L’article SQL Server: Disabling FileTable/FileStream on an AlwaysOn high-availability instance est apparu en premier sur dbi Blog.

Reference: https://support.microsoft.com/en-us/servicing/sql/sql-server-2025/cumulative-update/kb5084896-cu5

I built a lab on CU5 and on later builds up to CU9 and I spent some time measuring. Most of the documentation is confirmed. The sentence that says what the option actually changes, is not: on my lab, under the conditions described below I could not reproduce it. This post is what I measured with the protocol so that you can run it yourself and tell me if you get something different.

Why would released locks still cost memory?

A lock is not one structure in memory, it is two. The lock block describes the locked resource: this row, this page, this table. The lock owner block describes who holds it and in which mode. Three sessions reading the same row give one lock block and three lock owner blocks. sys.dm_tran_locks returns one row per owner block: the resource_* columns describe the lock block, the request_* columns the owner block and lock_owner_address is literally its address in memory.

The documentation gives 96 bytes per lock: 64 bytes for the lock block, plus 32 bytes per owner block. On my lab, a row lock costs about 195 bytes all included, the clerk’s pages divided by the number of locks held: twice the documented figure. Part of the gap is known. The 64 bytes are the 32-bit size of the lock block: on 64-bit, Kalen Delaney gives 128 bytes. That leaves about 67 bytes for the owner block, the share of the intent lock on the page and the overhead of the object store. Keep 200 bytes in mind. Ten million row locks are 2 GB.

These structures are not database pages. They are stolen memory taken from a pool owned by the lock manager. This pool is visible as the OBJECTSTORE_LOCK_MANAGER memory clerk in sys.dm_os_memory_clerks (one clerk per NUMA node, named Lock Manager : Node 0…) and as the Lock Memory (KB) performance counter. The locks documentation says 2500 lock structures. On my lab, Lock Blocks Allocated reports 3050 right after a restart, on SQL Server 2022 as on 2025.

Reference : https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-the-locks-server-configuration-option?view=sql-server-ver17

Here is what I have :

SELECT
    cntr_value AS lock_blocks_allocated
FROM sys.dm_os_performance_counters
WHERE object_name LIKE N'%:Memory Manager%'
  AND counter_name = N'Lock Blocks Allocated';

Now the important part. When a transaction commits or rolls back, its locks are released. The lock structures are not returned to SQLOS. They stay in the pool, ready to be reused by the next lock request. This is a cache and the reason is simple: allocating memory for every lock acquisition would be expensive and lock acquisition is on the hot path of every query.

The numbers to remember

Four numbers. The documentation expresses the percentages against the “SQLOS committed memory”. On my lab, they behave as percentages of committed_target_kb in sys.dm_os_sys_info, the Target Server Memory (KB) counter: what SQLOS is allowed to use, not what it has used so far. Test 1 below shows why I say so.

Two remarks. Lock escalation exists precisely to keep lock memory small: 5000 row locks cost about 1 MB, one table lock costs 200 bytes. And the 60 % limit is not a cache limit: it is the maximum memory the lock manager may use, cache included.

What was the problem before CU5?

The cache could grow up to the 60 % limit. It was given back slowly, by a background task and that task could be slow enough for the cache to look permanent.

The typical trigger is a workload that acquires a very large number of locks at once. The documentation mentions large concurrent query workloads with lock escalation disabled at the instance level (trace flag 1211). But you do not need a trace flag for that. ALTER TABLE … SET (LOCK_ESCALATION = DISABLE) on a big table, a reporting query in REPEATABLE READ or SERIALIZABLE scanning millions of rows, a purge that could not escalate because another session held an incompatible lock on the table: any of these can push lock memory to several GB.

After the transaction ends, the lock structures stay in the cache. The buffer pool, the plan cache and every other cache have to live with what is left. Page life expectancy drops, physical reads increase, and nothing in the current workload explains it, because the workload that caused it is over.

The symptom is easy to recognize once you know it. An idle instance, sys.dm_tran_locks almost empty and OBJECTSTORE_LOCK_MANAGER reporting GB. On an instance with a 128 GB target, the cache can hold up to about 77 GB.

What the documentation says CU5 changes

The lock manager cache now has a ceiling and you can configure it:

EXECUTE sp_configure 'show advanced options', 1;
RECONFIGURE;

EXECUTE sp_configure 'max lock manager cache memory (%)', 20;
RECONFIGURE;

The server must be restarted before the setting can take effect. The Server configuration options page lists it as A, RR (advanced, restart required): https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/server-configuration-options-sql-server?view=sql-server-ver17

What the option limits is the cache, not the locks.

As stated in the documentation:

Lock manager memory can still grow up to 60 percent of the SQLOS committed memory if required by the workload. However, when locks are released, memory is freed rather than cached if the lock manager cache already reached its configured limit.

Reference : https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/max-lock-manager-cache-memory-configuration-option?view=sql-server-ver17

It says two things. A transaction may still take up to 60 % of the memory in locks: true (test 1). And when locks are released while the cache is already at its limit, the memory is freed instead of cached: this is the sentence I could not reproduce (tests 2 and 3).

The probe

;WITH lm AS
(
    SELECT SUM(pages_kb) AS lock_manager_kb
    FROM sys.dm_os_memory_clerks
    WHERE type = N'OBJECTSTORE_LOCK_MANAGER'
),
si AS
(
    SELECT committed_kb, committed_target_kb
    FROM sys.dm_os_sys_info
),
pc AS
(
    SELECT
        MAX(CASE WHEN counter_name = N'Lock Blocks'                THEN cntr_value END) AS lock_blocks_in_use,
        MAX(CASE WHEN counter_name = N'Lock Blocks Allocated'      THEN cntr_value END) AS lock_blocks_allocated,
        MAX(CASE WHEN counter_name = N'Lock Memory (KB)'           THEN cntr_value END) AS lock_memory_kb,
        MAX(CASE WHEN counter_name = N'Database Cache Memory (KB)' THEN cntr_value END) AS database_cache_kb,
        MAX(CASE WHEN counter_name = N'Free Memory (KB)'           THEN cntr_value END) AS free_memory_kb
    FROM sys.dm_os_performance_counters
    WHERE object_name LIKE N'%Memory Manager%'
)
SELECT
    SYSDATETIME()  AS measured_at,
    lm.lock_manager_kb / 1024  AS lock_manager_mb,
    si.committed_kb / 1024  AS committed_mb,
    si.committed_target_kb / 1024 AS target_mb,
    CAST(100.0 * lm.lock_manager_kb / si.committed_target_kb AS DECIMAL(5, 1)) AS lock_manager_pct,
    pc.lock_blocks_in_use,
    pc.lock_blocks_allocated,
    pc.lock_blocks_allocated - pc.lock_blocks_in_use AS lock_blocks_cached,
    pc.database_cache_kb / 1024 AS buffer_pool_mb,
    pc.free_memory_kb / 1024 AS free_mb
FROM lm
CROSS JOIN si
CROSS JOIN pc;

How to read it:

• lock_manager_mb is the whole lock manager memory: locks in use plus cached structures. On an idle instance, it is the size of the cache. This is the measure that matters.
• committed_mb is what SQLOS has taken from the operating system so far. target_mb is what it is allowed to take. On an instance that has been running for a while, the two are equal. On a freshly restarted lab, they are far apart and nothing in the percentages makes sense until they meet.
• lock_manager_pct is the clerk against the target. The 20 % and the 60 % are read on this column.
• buffer_pool_mb and free_mb show who pays for the cache, and who gets the memory back.
• lock_blocks_in_use, lock_blocks_allocated and their difference lock_blocks_cached are the performance counters on lock blocks. They are coherent on a freshly started instance (3050 allocated, 0 in use) and stop refreshing after a lock storm on every 2025 build I tested: 6 624 047 allocated blocks reported while the clerk held 1 MB. They are here so that you can see it. Do not use them as evidence, sys.dm_tran_locks is the truth for locks held, the clerk for memory.

The lab

SQL Server 2025 Developer Edition, CU5 (17.0.4045.5) first, then CU9 (17.0.5005.3), on virtual machines of a Proxmox lab and on a Nutanix environment. One SQL Server 2022 instance for comparison, in test 5. The SQLOS target is 4096 MB unless stated otherwise (2048 MB in test 4), with min server memory and max server memory pinned to the same value.

A database with one narrow table. Lock escalation is disabled on the table: we want millions of row locks, not one table lock. This is the scenario the documentation describes at table level instead of trace flag 1211:

CREATE DATABASE LockCacheDemo
GO
ALTER DATABASE LockCacheDemo SET RECOVERY SIMPLE
GO
USE LockCacheDemo
GO
CREATE TABLE dbo.T_Locks
(
    id     INT      NOT NULL,
    filler CHAR(20) NOT NULL CONSTRAINT DF_T_Locks_filler DEFAULT ('x'),
    CONSTRAINT PK_T_Locks PRIMARY KEY CLUSTERED (id)
)
GO
INSERT INTO dbo.T_Locks WITH (TABLOCK) (id)
SELECT value
FROM GENERATE_SERIES(1, 30000000)
GO
ALTER TABLE dbo.T_Locks SET (LOCK_ESCALATION = DISABLE)
GO

Thirty million rows, about 1 GB. Nothing in this block touches the lock manager: the TABLOCK on the insert takes one table lock to load fast. This block only provides lockable rows and removes the mechanism that would stop us from locking them one by one.

The workload (session A):

USE LockCacheDemo
GO
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
GO
DECLARE @rows INT = 1000000;

BEGIN TRANSACTION

SELECT COUNT(*) AS rows_locked
FROM (
    SELECT TOP (@rows) id
    FROM dbo.T_Locks WITH (ROWLOCK)
    ORDER BY id
) AS x

-- Leave the transaction open. Measure from session B. Then:
-- ROLLBACK TRANSACTION

Three different settings:

From a second session (session B): did we get row locks and did escalation stay out?

SELECT resource_type, request_mode, COUNT(*) AS lock_count
FROM sys.dm_tran_locks
WHERE request_session_id = 65   -- The session holding the locks
GROUP BY resource_type, request_mode
ORDER BY lock_count DESC;

One shared lock per row (KEY, the row of a clustered index). One intent lock per page, about 245 rows per page. One intent lock on the table and no OBJECT S: no escalation.

Second check, the cost of a lock. Measured on a restarted instance each time so that nothing was cached before the run:

DECLARE @spid    INT    = 65;  -- session A
DECLARE @idle_kb BIGINT = 0;   -- clerk before the transaction

SELECT
    l.lock_count,
    l.key_locks,
    c.lock_manager_kb / 1024                                  AS lock_manager_mb,
    (c.lock_manager_kb - @idle_kb) * 1024.0 / l.lock_count    AS bytes_per_structure,
    (c.lock_manager_kb - @idle_kb) * 1024.0 / l.key_locks     AS bytes_per_row_lock
FROM (
    SELECT COUNT_BIG(*)                                            AS lock_count,
           SUM(CASE WHEN resource_type = N'KEY' THEN 1 ELSE 0 END) AS key_locks
    FROM sys.dm_tran_locks
    WHERE request_session_id = @spid
) AS l
CROSS JOIN (
    SELECT SUM(pages_kb) AS lock_manager_kb
    FROM sys.dm_os_memory_clerks
    WHERE type = N'OBJECTSTORE_LOCK_MANAGER'
) AS c;

For example :

Four runs, the same figure: about 195 bytes per row lock, everything included. Twice the 96 bytes of the documentation (64 for the lock block, 32 per owner block).

Test 1: The 60 % wall has not moved

The name of the option invites a misunderstanding: it is not a limit on locks. Option at its default of 20 and @rows = 70 000 000 far beyond what 60 % of the target can hold (Target at 4968 MB for this run). The lock manager climbs.

The probe shows 20 %, then 24 % without any escalation (the table option prevents it), then 40 %, 50 % and the statement dies:

Severity 19: the statement is terminated and the transaction is rolled back. The same message lands in the SQL Server error log and in the Windows application log.

Option at its default of 20 and @rows = 70 000 000, more than the 30 million rows of the table and far beyond what 60 % of the target can hold (Target at 4968 MB for this run): the wall is at about 16 million row locks.

This test also settled which memory figure the percentages refer to. On the last probe before the error, the clerk was at 2979 MB: 60 % of the 4968 MB target, and 70 % of the 4233 MB SQLOS had committed at that moment. A 60 % limit on the committed memory would have fired about 440 MB earlier, at 2540 MB. The 60 % limit, and with it the 20 % one, are percentages of the target, not of the memory committed so far. This is why the probe divides by committed_target_kb.

Test 2: Releasing the locks frees nothing

This is the test I wrote this post for. Option at 20. Target at 4096 MB for this run, so the cache ceiling is 819 MB. Nine million locks:

buffer_pool_mb is the Database Cache Memory (KB) counter: the data and index pages in memory. The 349 MB are the pages of the nine million rows the SELECT had to read to lock them, nothing else touched the buffer pool. sys.dm_tran_locks is empty, the session holds nothing and the clerk has not moved. 1677 MB of lock structures are sitting in the cache, twice the configured ceiling. The ROLLBACK did not free anything and neither did the minutes after it.

I ran this on CU5 and on CU9, with 5, 6, 9 and 12 million locks with targets of 2 048 and 4 096 MB. Same result every time. The last run, 12 million locks on CU9 left 2193 MB in the cache, 54 % of the target and they were still there hours later.

Test 3: Later transactions free nothing either

There is a second way to read the sentence of the documentation. Not the release of the storm but the releases that follow: once the cache is above its limit, every structure released by a later transaction should be freed instead of cached and the cache would erode with the lock activity of the workload, down to 20 %. “Large concurrent query workloads” as the documentation puts it.

A first attempt with UPDATE TOP (1000000) … SET filler = ‘y’ and a COMMIT, repeated, lowered the clerk by a few MB per pass. Not conclusive: without a ROWLOCK hint, the engine is free to take page locks for an update of this size and then only a few thousand structures travel through the cache at each pass.

So the proper version, with row locks guaranteed and verified. Cache at 40.9 % of the target after a storm, option at 20, and short transactions in a loop:

USE LockCacheDemo
GO

SET NOCOUNT ON;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;

DECLARE @i INT = 0, @n INT;

WHILE @i < 100
BEGIN
    BEGIN TRANSACTION;
    SELECT @n = COUNT(*)
    FROM (SELECT TOP (200000) id
          FROM dbo.T_Locks WITH (ROWLOCK)
          WHERE id > (@i % 50) * 200000
          ORDER BY id) AS x;
    COMMIT;            -- 200 000 structures released
    SET @i += 1;
END

One hundred transactions, 200,000 row locks each, verified in sys.dm_tran_locks while the loop ran: twenty million lock releases, more than four times the 4.6 million structures sitting above the ceiling (858 MB).

The clerk did not move. Twenty million releases while the cache was more than twice its limit and the memory stayed. Whatever “when locks are released, memory is freed rather than cached” describes, it is not something I can observe with row locks, on CU5 or CU9, with the option at its default.

Test 4: What does shrink the cache

Two things did, during the week. None of them is the sentence of the documentation. Memory demand, once. Target at 2 048 MB, option at 20, six million locks, ROLLBACK: 1 117 MB in the cache, 54.6 %. Then a NOLOCK scan of a 17 GB table which takes one schema stability lock and nothing else but needs every page in the buffer pool:

The scan pushed committed_mb up to the target at that moment the buffer pool still wanted pages, SQLOS had nothing left to take from the operating system and it took from the lock manager: 785 MB given back, the clerk under the ceiling. Not one lock released in between. That looked like the mechanism. It was not reproducible either: on CU9 with the cache at 2 193 MB and the instance already at its 4 096 MB target, the same 17 GB scan left the cache at 2 190 MB. sys.dm_os_ring_buffers tells why: the memory broker that governs the lock manager cache (MEMORYBROKER_FOR_CACHE, which is where the object store lives, not MEMORYBROKER_FOR_STEAL) kept sending it a target of 2 809 MB, 68 % of the target with SHRINK notifications of a few MB. There is no 20 % anywhere in the broker’s arithmetic.

DBCC FREESYSTEMCACHE, instantly. The documentation only shows ‘ALL’, which releases the unused entries of every cache, plan cache included, at the price of recompiling every plan afterwards. It also says that most caches can be freed individually. The lock manager cache is one of them:

SELECT name, pages_kb / 1024 AS mb
FROM sys.dm_os_memory_clerks
WHERE name LIKE N'Lock Manager%'; 

DBCC FREESYSTEMCACHE('Lock Manager : Node 0');

Before :

After :

Probe right after, on the 1677 MB cache left by test 3: 8 MB in the clerk, and the memory back in free_mb. The cache is releasable at any time in one instruction. What is missing on these builds is not the ability to release it, it is the trigger.

Test 5: SQL Server 2022 does the same without the option

If the option changed something, the comparison with a version that does not have it should show it. Same table, same workload, @rows = 70000000 for the wall and a value under the wall for the clean ROLLBACK, on SQL Server 2022 and on SQL Server 2025 CU5, CU9 with the option at 20:

Two things in this table. The background release exists on 2022, without any option, and it brings the cache down to 17.6 %, which is where the 20 % ceiling of 2025 would put it. And 2025 is not faster than 2022 at giving the memory back.

What the tests say

Put together, on SQL Server 2025 CU5 and CU9 with one big transaction and then normal lock activity:

• Lock memory, cache included, can grow up to 60 % of the target. Beyond that, error 1204. Confirmed and unchanged by the option.
• When the storm is released, nothing is freed. The structures go to the cache above the configured ceiling and stay there for hours.
• The lock activity that follows does not free them either: twenty million row-lock releases above the ceiling, no measurable change.
• The cache does come down, slowly, through the generic SQLOS mechanisms (background release, memory broker rebalancing when the instance is at its target), the same ones SQL Server 2022 has. On 2025, they are not faster, and the 20 % figure does not appear in the broker’s targets.
• DBCC FREESYSTEMCACHE(‘Lock Manager : Node N’) empties the cache immediately on any version I tried.

So the documented behavior, “when locks are released, memory is freed rather than cached if the lock manager cache already reached its configured limit”, is not something I could reproduce. Not on CU5, not on CU9, not with the release of the big transaction, not with twenty million releases afterwards. The option is there, it accepts 20 to 60, value_in_use says 20 and the cache sits around 40% for as long as I care to watch.

Two caveats, because this is a lab and not a support case. One access pattern: a single transaction holding millions of row locks then released. Virtual machines with 2 and 4 GB targets. Developer Edition. It is possible that the mechanism needs a condition I did not create and that “large concurrent query workloads” means something more specific than what I ran. If you reproduce the protocol above and see the cache come back under 20 % after a release, I want to hear about it. If you do not, you have the same question I have.

A summary

• Released locks are not freed, they are cached by the lock manager. A row lock costs about 195 bytes all included.
• The thresholds (24 % escalation, 60 % hard limit, 20 % cache ceiling) are read against committed_target_kb, not committed_kb. On a lab, set min and max server memory before measuring anything.
• CU5 adds max lock manager cache memory (%): 20 by default, 20 to 60 allowed, a restart to take effect (is_dynamic = 0).
• The 60 % wall and error 1204 are unchanged by the option.
• On CU5 and CU9, releasing the locks does not bring the cache back under the ceiling and neither do twenty million later releases. The documented behavior was not reproducible on my lab.
• The cache does shrink through the generic SQLOS mechanisms slowly as on SQL Server 2022 and the memory broker targets it well above 20 %.
• DBCC FREESYSTEMCACHE(‘Lock Manager : Node N’) empties it instantly (on any version I tried). It is the remedy that works.
• The lock block performance counters stop refreshing on these builds. Use the clerk and sys.dm_tran_locks.

Thank you. Amine Haloui

L’article SQL Server 2025 CU5: max lock manager cache memory (%): what I could reproduce and what I could not est apparu en premier sur dbi Blog.

To centralise Oracle naming resolution, you can use an LDAP server. You can use the commercial products from Oracle like Oracle Internet Directory (OID) or Oracle Universal Directory (OUD). Alternatively, you can also use OpenLDAP from your Linux distribution. I have used this successfully with customers for many years.

I now wanted to build a new test environment for myself on an Enterprise Linux 9 system (RedHat, Oracle Linux). But then I noticed that the opendap server is no longer available. After some internet searching, I found the following blog post that describes how to install OpenLDAP on EL9: https://seanstuber.com/2019/12/29/setting-up-openldap-on-oracle-linux-8-for-tns-name-resolution/

But is there no way to do this with on-board tools? A further search then revealed that RedHat has switched to 389-ds-base (https://www.port389.org/). So I want to try it with this native tool. And as I fear, almost everything is different here again… To spare you this try-and-error, I’ll give you the commands here on how to get an executable environment.

Installation and configuration

First of all, install the software:

[root@ol9b ~]# dnf install -y 389-ds-base openldap-clients

Caution! I stongly recommend to do a dnf upgrade before. I had the problem that you couldn’t log in with ssh afterwards and the ssh daemon couldn’t be restarted.

OpenSSL version mismatch. Built against 30000070, you have 30500080

At least a dnf upgrade "openssh*" was required and restart of sshd.

Afterwards, you can configure and start the LDAP server. We will use “orcladmin” as administrative user and “dbi” as instance name.

[root@ol9b ~]# dscreate interactive
Install Directory Server (interactive mode)
===========================================

Enter system's hostname [ol9b]:

Enter the instance name [ol9b]: dbi

Enter port number [389]:

Create self-signed certificate database [yes]:

Enter secure port number [636]:

Enter Directory Manager DN [cn=Directory Manager]: cn=orcladmin

Enter the Directory Manager password:
Confirm the Directory Manager Password:

Choose whether mdb or bdb is used. [bdb]:

Enter the database suffix (or enter "none" to skip) [dc=ol9b]: none

Do you want to start the instance after the installation? [yes]:

Are you ready to install? [no]: yes
Starting installation ...
Validate installation settings ...
Create file system structures ...
Create self-signed certificate database ...
Perform SELinux labeling ...
Perform post-installation tasks ...
Completed installation for instance: slapd-dbi

[root@ol9b ~]#  systemctl status dirsrv@dbi
● dirsrv@dbi.service - 389 Directory Server dbi.
     Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/dirsrv@.service.d
             └─custom.conf
...

After configuration, the daemon is already started and the service is also set to enabled. If you use a firewall on your server, open the ldap ports:

[root@ol9b ~]# firewall-cmd --permanent --add-service=ldap
[root@ol9b ~]# firewall-cmd --permanent --add-service=ldaps
[root@ol9b ~]# firewall-cmd --reload

Now we have to extend the schema with the extensions for Oracle naming resolution. Since Oracle 26ai (23.26), a schema file is available in $ORACLE_HOME/network/ldap/schema/openldap/orclnetnaming.schema; This file has to be converted to an ldif file for the ldapmodify tool. From:

attribute ( 2.16.840.1.113894.7.1.1
  NAME 'orclVersion'
  EQUALITY octetStringMatch
  SYNTAX '1.3.6.1.4.1.1466.115.121.1.40' )

attribute ( 2.16.840.1.113894.7.1.2
  NAME 'orclOracleHome'
  EQUALITY octetStringMatch
  SYNTAX '1.3.6.1.4.1.1466.115.121.1.40' )
...

to:

dn: cn=schema
changetype: modify
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.1
  NAME 'orclVersion'
  EQUALITY octetStringMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.2
  NAME 'orclOracleHome'
  EQUALITY octetStringMatch
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
...

(the full version you will find at the end of this post)

And now you can load this schema extensions

[root@ol9b ~]# ldapmodify -x -H ldap://ol9b:389 -D "cn=orcladmin" -W -f orclnetnaming.ldif
Enter LDAP Password:
modifying entry "cn=schema"

We will create the DN dbi-services,dn=com

[root@ol9b ~]# dsconf dbi backend create --suffix="dc=com" --be-name="com" --create-suffix
The database was sucessfully created

[root@ol9b ~]# ldapadd -x -H ldap://ol9b:389 -D "cn=orcladmin"  -W <<EOD
dn: dc=dbi-services,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: Oracle Internet Directory
dc: dbi-services

dn: cn=OracleContext,dc=dbi-services,dc=com
objectclass: orclContext
cn: OracleContext
EOD
Enter LDAP Password:
adding new entry "dc=dbi-services,dc=com"

adding new entry "cn=OracleContext,dc=dbi-services,dc=com"

Oracle tools need the so-called “anonymous bind” so that they can query the data without logging in. We still have to activate this.

[root@ol9b ~]# ldapmodify -x -H ldap://ol9b:389 -D "cn=orcladmin"  -W <<EOD
 dn: dc=dbi-services,dc=com
changetype: modify
add: aci
aci: (targetattr = "*")(version 3.0; acl "Anonymous read access"; allow (read, search, compare) userdn = "ldap:///anyone";)
EOD

The configuration is now complete, and we can insert the first sqlnet alias

[root@ol9b ~]# ldapadd -x -H ldap://ol9b:389 -D "cn=orcladmin"  -W <<EOD
dn: cn=DB1970,cn=OracleContext,dc=dbi-services,dc=com
objectclass: top
objectclass: orclNetService
cn: DB1970
orclNetDescString: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ltp14s-i)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=DB1970A.erfinder)))
EOD
Enter LDAP Password:
adding new entry "cn=DB1970,cn=OracleContext,dc=dbi-services,dc=com"

Hint: this is the syntax of OpenLdap. As oracle user, use /usr/bin/ldapadd, or adapt the syntax for Oracle’s ldapadd.

Testing

Now it gets exciting: does the Oracle name resolution work?

Probably we have to adapt our ldap.ora and sqlnet.ora files. For the test, I will create a temporary TNS_ADMIN directory.

[oracle@ltp14 ~]# mkdir /tmp/ldap
[oracle@ltp14 ~]# export TNS_ADMIN=/tmp/ldap

[oracle@ltp14 ~]# echo 'DIRECTORY_SERVERS= (ol9b:389:636)
DEFAULT_ADMIN_CONTEXT = "dc=dbi-services,dc=com"
DIRECTORY_SERVER_TYPE = OID' >$TNS_ADMIN/ldap.ora

[oracle@ltp14 ~]# echo 'NAMES.DEFAULT_DOMAIN=dbi-services.com
NAMES.DIRECTORY_PATH=(LDAP)' >sqlnet.ora

[oracle@ltp14 ~]# tnsping DB1970

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 18-SEP-2026 13:48:26

Copyright (c) 1997, 2019, Oracle.  All rights reserved.

Used parameter files:
/u00/app/oracle/network/admin/ldapdbi/sqlnet.ora

Used LDAP adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ltp14s-i)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=DB1970A.erfinder)))
OK (0 msec)

And YES! it works 🙂

Disclaimer: I’m not the expert of LDAP. I wanted to be able to quickly set up an LDAP server for Oracle name resolution. The setup certainly still has some optimization potential. Feel free to adapt it for your needs.

Annex

orclnetnaming.ldif (converted from 26ai $ORACLE_HOME/network/ldap/schema/openldap/orclnetnaming.schema)

dn: cn=schema
changetype: modify
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.1 NAME 'orclVersion' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.2 NAME 'orclOracleHome' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.3 NAME 'orclSystemName' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.4 NAME 'orclServiceType' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.5 NAME 'orclSid' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.7.1.6 NAME 'orclProductVersion' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.1 NAME 'orclNetSourceRoute' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.2 NAME 'orclNetLoadBalance' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.3 NAME 'orclNetFailover' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.4 NAME 'orclNetSdu' EQUALITY numericStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.36 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.5 NAME 'orclNetServer' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.6 NAME 'orclNetServiceName' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.7 NAME 'orclNetInstanceName' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.8 NAME 'orclNetHandlerName' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.9 NAME 'orclNetParamList' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.10 NAME 'orclNetAuthenticationType' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.11 NAME 'orclNetAuthParams' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.12 NAME 'orclNetDescName' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.13 NAME 'orclNetDescString' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.14 NAME 'orclNetAddressString' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.15 NAME 'orclNetProtocol' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.16 NAME 'orclNetShared' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.17 NAME 'orclNetAddrList' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.18 NAME 'orclNetProtocolStack' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.19 NAME 'orclNetDescList' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.20 NAME 'orclNetConnParamList' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: attributeTypes
attributeTypes: ( 2.16.840.1.113894.3.1.21 NAME 'orclNetAuthenticationService' EQUALITY octetStringMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.7.2.3 NAME 'orclContext' SUP top MAY ( cn $   orclNetDescName ) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.7.2.6 NAME 'orclSchemaVersion' SUP top MUST ( cn $   orclProductVersion ) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.3.2.1 NAME 'orclNetAddress' SUP top MUST (cn) MAY ( orclNetAddressString $ orclNetProtocol $ orclNetShared $ orclVersion $description) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.3.2.2 NAME 'orclNetAddressList' SUP top MUST (cn)
     MAY ( orclNetAddrList $ orclNetSourceRoute $ orclNetLoadBalance $ orclNetFailover $ orclNetShared $ orclVersion $ description ) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.3.2.3
     NAME 'orclNetDescription' SUP top
     MUST (cn)
     MAY ( orclVersion  $ orclNetSourceRoute  $  orclNetSdu $ orclNetServer $ orclNetServiceName $ orclNetInstanceName $ orclNetHandlerName $ orclNetParamList $ orclNetAuthenticationType $ orclNetAuthParams $ orclNetShared $ orclNetAddrList $  orclNetProtocolStack $ orclNetConnParamList $ orclNetAuthenticationService $ orclNetLoadBalance $   orclNetDescList $ orclNetFailover $ orclNetShared $ orclNetAddrList ) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.3.2.30
     NAME 'orclNetDescriptionAux1' SUP top
     AUXILIARY MUST (cn)
     MAY ( orclVersion  $ orclNetSourceRoute  $  orclNetSdu $ orclNetServer $ orclNetServiceName $ orclNetInstanceName $ orclNetHandlerName $ orclNetParamList $ orclNetAuthenticationType $ orclNetAuthParams $ orclNetShared $ orclNetAddrList $  orclNetProtocolStack $ orclNetConnParamList $ orclNetAuthenticationService $ orclNetLoadBalance $   orclNetDescList $ orclNetFailover $ orclNetShared $ orclNetAddrList ) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.3.2.4
     NAME 'orclNetDescriptionList' SUP top
     MUST (cn)
     MAY ( orclNetLoadBalance $   orclNetDescList $ orclNetFailover $ orclNetShared $ orclNetAddrList ) )
-
add: objectClasses
objectClasses: ( 2.16.840.1.113894.3.2.5
     NAME 'orclNetService' SUP top
     MUST (cn)
     MAY ( orclNetServiceName $ orclNetDescString $ orclNetDescName $ orclNetSourceRoute ) )

L’article Setup an LDAP server for Oracle naming on Enterprise Linux 9 est apparu en premier sur dbi Blog.

When a company still refuses to put its sensitive data in the cloud, the reason usually isn’t cost or performance: it’s data sovereignty. In the cloud, someone else is the administrator of the machine; therefore the provider can, in theory, read the data files, see the data being used in memory, or read the backups. The threat model is no longer the external attacker but the privileged insider hosting the database.

As covered previously on my blog Beyond TDE and TLS: Bridging the Data Security Governance Gap in Lower Environments, various encryption methods can protect you. For example, TLS protects data in transit and TDE protects it at rest, but as soon as the engine runs a query, it handles plaintext in memory. So there are three states to protect: at-rest, in-transit, in-use. The gap this article focuses on is the last one.

That is exactly what Always Encrypted (SQL Server) and Queryable Encryption (MongoDB) target. The common principle: encryption and decryption happen client-side, in the driver; the keys never reach the engine. The data stays encrypted at rest, in transit, and during processing. The DBA, the cloud operator, the hypervisor admin: all of them see only cyphertext.

That leaves one question: if the engine sees only cyphertext, how does it answer a WHERE condition? Both database engines do answer it, but through different technical means.

SQL Server: queryability lives in the cyphertext

To demonstrate all this, let’s start by creating a table with two columns, Salary and Department, encrypted deterministically on one side and randomized on the other:

DROP TABLE IF EXISTS dbo.Employees;
CREATE TABLE dbo.Employees (
  Id         INT IDENTITY(1,1) PRIMARY KEY,
  LastName   NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  FirstName  NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  DeptDet    NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be DETERMINISTIC
  DeptRand   NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be RANDOMIZED
  SalaryDet  INT NOT NULL, -- will be DETERMINISTIC
  SalaryRand INT NOT NULL  -- will be RANDOMIZED
);
GO

INSERT INTO dbo.Employees (LastName, FirstName, DeptDet, DeptRand, SalaryDet, SalaryRand) VALUES
('Martin',  'Alice', 'Sales', 'Sales', 55000, 55000),
('Dubois',  'Bob',   'Sales', 'Sales', 48000, 48000),
('Bernard', 'Chloe', 'Sales', 'Sales', 52000, 52000),
('Petit',   'David', 'IT',    'IT',    72000, 72000),
('Durand',  'Emma',  'IT',    'IT',    68000, 68000);

Five rows: three Sales, two IT. The BIN2 collation is required by Always Encrypted (link to documentation), and the master key lives outside the database (Key Vault, certificate store, or HSM).

Column encryption isn’t done in T-SQL, because the engine doesn’t have the keys. It’s driven from the client (here in PowerShell), declaring for each value a deterministic column and its randomized twin:

Import-Module SqlServer -MinimumVersion 22.0.59
$sqlConnectionString = "Data Source=.\LAB2025;Initial Catalog=AEDEMO;Integrated Security=True;Encrypt=False;Trust Server Certificate=False"
$smoDatabase = Get-SqlDatabase -ConnectionString $sqlConnectionString

$encryptionChanges  = @()
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptDet    -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptRand   -EncryptionType Randomized    -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryDet  -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryRand -EncryptionType Randomized    -EncryptionKey "CEK1"

Set-SqlColumnEncryption -ColumnEncryptionSettings $encryptionChanges -InputObject $smoDatabase

In this example, I’m working on my 2025 SQL Server instance, on the AEDEMO database, using the column encryption key CEK1 I created beforehand (itself protected by a column master key stored outside the database).

We check that the engine sees the right type per column:

SELECT c.name, c.encryption_type_desc
FROM sys.columns c
WHERE c.object_id = OBJECT_ID('dbo.Employees');

The deterministic mechanism works like this: same plaintext, same cyphertext (an injective function). On an Always Encrypted-enabled connection with Parameterization for Always Encrypted and parameters for the predicates (never literals), equality works:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptDet AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptDet = @d
GROUP BY DeptDet;

The driver encrypts @d with the same key, the server finds the encrypted values that are identical to the parameter, and the result is correct.

You pay for it in three ways…

First weakness: deterministic encryption can cause a data leak. You don’t need the keys to see it. Just connect without Always Encrypted and read the table: the encrypted columns come out as raw binary.

SELECT FirstName, DeptDet, DeptRand FROM dbo.Employees;

Look at the DeptDet column: Alice, Bob, and Chloe share exactly the same blob, and David and Emma share another. Two distinct values across five rows. The adversary doesn’t know what 0x012536… means, but reads the structure: two departments, one with three people, the other with two, and who goes with whom. The DeptRand column, on the other hand, shows five all-different blobs: nothing to read.

It’s harmless on five rows; it isn’t on a real table. On a low-cardinality column (region, sex, status), the distribution of blobs can be compared to a known distribution, and frequency analysis often reconstructs the plaintext. Microsoft’s documentation puts it bluntly: an unauthorized user can guess information by examining patterns, especially when the set of possible values is small.

Second weakness: randomized encryption blocks every query. With randomized encryption, the driver adds a fresh random value to each cell before encrypting, so the same input produces a different cyphertext every time. Therefore, the leak is gone but so is the query:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptRand AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptRand = @d
GROUP BY DeptRand;  

Msg 33277, Level 16, State 2, Line 6
Encryption scheme mismatch for columns/variables ‘DeptRand’, ‘@d’. The
encryption scheme for the columns/variables is (encryption_type =
‘RANDOMIZED’, …) and the expression near line ‘6’ expects it to be
DETERMINISTIC, or RANDOMIZED, a BIN2 collation for string data types,
and an enclave-enabled column encryption key, or PLAINTEXT.

The server can no longer compare, since the same plaintext produces a different cyphertext on every row. We gained confidentiality and lost the query. It’s all or nothing.

Third weakness: no range, even with deterministic. Byte equality says nothing about order:

DECLARE @s INT = 55000;
SELECT FirstName FROM dbo.Employees WHERE SalaryDet < @s;

WHERE SalaryDet < @s fails even though the column is deterministic. Sorting, BETWEEN, LIKE: out of reach for Always Encrypted alone.

The verdict is clear: bare Always Encrypted means equality, or nothing (=, IN, GROUP BY, and DISTINCT supported).

MongoDB: queryability lives in a protocol

MongoDB’s Queryable Encryption makes optimal use of randomized encryption. On the server side, everything is Randomized-encrypted, as the documentation explains: the server has no knowledge of the data it processes. The encrypted view (a client connected without the keys) confirms it: even the three Sales employees come out with all-different BinData.

No frequency leak, unlike SQL Server’s deterministic encryption. With the keys, the same find returns the plaintext:

The technical mechanism behind this lies in the collection’s declaration. Each encrypted field carries a queryType, its parameters, and a distinct key (keyId), one Data Encryption Key per field, which is mandatory:

And queries with equality tests, range, and even equality on a field declared as range all work:

On the server side, MongoDB then maintains encrypted index structures, and for each query the driver generates cryptographic tokens that the server checks against those structures without ever seeing the plaintext. This is a searchable encryption scheme. Range is available in GA, with no special hardware.

The real difference comes down to one thing: the driver

On the SQL Server side, the driver’s work stays thin: it encrypts the parameters, rewrites the query, decrypts the results. Queryability itself is already carried by the cyphertext; the driver doesn’t have to handle it. On the MongoDB side, the driver carries the whole protocol: it generates the tokens checked against the encrypted indexes.

Once the collection is in place, find({ department: "Sales" }) is written like a normal query and the driver handles the encryption on its own. Each encrypted field needs its own key (Data Encryption Key). The master key must be pinned, otherwise orphaned keys return an HMAC validation failure error. And the encryption API is only available from a client created as encrypted, so not from a standard Compass connection, for example.

Summary

What if you wanted range while staying on SQL Server?

This is where secure enclaves come in. The engine delegates the computation to an enclave: a protected memory region where the data is decrypted and processed in the clear, out of reach, including from the machine’s administrator. This is what unlocks range, LIKE, sorting, and in-place encryption: inside the enclave the server no longer compares encrypted bytes, it works on the plaintext. This gain has a price. It requires compatible hardware or secure virtualization, an attestation service to deploy and maintain, and keys configured for the enclave, which noticeably increases architectural complexity compared with classic Always Encrypted.

But it also changes the nature of the trust. Queryable Encryption rests on a cryptographic guarantee: the server cannot read, it’s a mathematical property. Enclaves rest on a hardware guarantee: you trust the CPU, and its attestation, to isolate the protected region.

There remains a third path, often mentioned: homomorphic encryption (FHE), which computes directly on the cyphertext without ever decrypting it. Elegant on paper, but out of the game for database search: the computational cost is massive and response time collapses as the volume grows. So the practical choice really does play out between the two worlds described here.

L’article SQL Server vs MongoDB: When the cloud is your adversary (Always Encrypted vs Queryable Encryption) est apparu en premier sur dbi Blog.

This is something we often see among our customers. If we assume that we carry out a migration project roughly every five years, this means we’re keeping databases that are taking up space for no reason.

We, however, are Green SQL Server DBA; we’ll look to manage it correctly.

The AUDIT

First of all, what do we need to analyse?

This does not apply to all databases, but only to those that are obsolete.

They can be categorised into two types:

• Databases Never Accessed
  • No application connections detected
  • No recent transactions or queries after a period (3 months for exemple)
• Abandoned Databases
  • Linked to cancelled projects
  • No activity and nobody tell it

To see databases never accessed and abandoned databases I use the DMV sys.dm_db_index_usage_stats to see if I have an access to the database (read or write):

SELECT d.name AS DatabaseName, MAX(ius.last_user_seek) AS LastSeek,MAX(ius.last_user_scan) AS LastScan,
    MAX(ius.last_user_lookup) AS LastLookup,

    (
        SELECT MAX(v.ActivityDate)
        FROM (VALUES
                (MAX(ius.last_user_seek)),
                (MAX(ius.last_user_scan)),
                (MAX(ius.last_user_lookup))
             ) v(ActivityDate)
    ) AS LastReadActivity,

    MAX(ius.last_user_update) AS LastWriteActivity,

    DATEDIFF(DAY,
        (
            SELECT MAX(v.ActivityDate)
            FROM (VALUES
                    (MAX(ius.last_user_seek)),
                    (MAX(ius.last_user_scan)),
                    (MAX(ius.last_user_lookup))
                 ) v(ActivityDate)
        ),
        GETDATE()
    ) AS DaysSinceLastRead,

    DATEDIFF(DAY,
        MAX(ius.last_user_update),
        GETDATE()
    ) AS DaysSinceLastWrite

FROM sys.databases d
LEFT JOIN sys.dm_db_index_usage_stats ius
    ON d.database_id = ius.database_id
WHERE d.database_id > 4  -- Exclude system databases
GROUP BY d.name
ORDER BY LastWriteActivity ASC;

We can see on the first databases in the list, that we have null everywhere… This means that the databases were not solicited since the last restart… Good candidate for unused databases!

This brings us to the next point.

The ANALISYS

If we can see that there is little activity on the database, we can, of course, carry out an initial analysis using criteria such as these:

< 30 days –> the database is still active

30–180 days –> the database is on hold pending to be dropped

> 180 days –> the database is a strong candidate to be dropped

Of course, it also depends on your environment – whether it’s dev, test or prod – and the criteria will vary accordingly.

The IMPACT

Like in my precedent post, we will have same impacts but not for queries of courses!

The first impact is the storage and the backup

If the database represents ~100 GB on the disk , it’s also 100 GB backed up unnecessarily.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 1 TB!

The third impact is on the maintenance plan

The following operations must also handle the unused database:

– CheckDB
– Backups

As the database is not used, the index rebuild/reorg & update Stats will be fast!

Each unused database extends the maintenance windows for the checkdb and backups…

The ADVISE

The first advice would be to plan well in advance.

Indeed, when creating or restoring a database, it is important to know its lifecycle.

Don’t hesitate to challenge the application owners on a deadline and make sure to add it to your calendar +1day.

The second advice is to set up a monthly monitoring process prior to Windows or SQL Server patch cycles so that you can identify these unnecessary databases.

And before dropping a database, I advice you to put the database offline and see if after some days somebody complaints…

In the majority of cases, you will not have reactions, believe me!

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will let the unused databases until the next migration.

A Green SQL Server DBA will check it periodically and manage it.

An unused database is like a car that never leaves the garage.
 It still costs money, requires maintenance, and consumes resources.
The Green SQL Server DBA notices it before it becomes waste…

Think about it and begin now to see to monitor your unused databases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #3 – The hidden cost of forgotten databases est apparu en premier sur dbi Blog.

Can this also be applied to PostgreSQL?

In short, yes, it is possible. Let’s see how.

Here is the list of the previous blog posts:

• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-2-3/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-rest-api-with-sql-server-2025-3-3/

Partitioning and filesystem

We will reuse the sqlpool ZFS pool created in the first part of this series.

We identify the 300 GB disk attached to the VM. In our case, it is /dev/sdb, backed by the sqlpool/pve/vm-307-disk-0 zvol on the Proxmox side:

lsblk

We create a single partition of type Linux filesystem:

sudo sgdisk -n 1:0:0 -t 1:8300 /dev/sdb

We format the partition with XFS, which is the most commonly recommended filesystem for PostgreSQL data directories:

sudo mkfs.xfs -L pgdata /dev/sdb1 -f

We verify the result:

ahi@pgl:~$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="pgdata" UUID="028afa2f-7bb3-4a40-92aa-91c1a33f18ae9" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="bd54f285-092e-4b1b-ba5e-6877f054fa7"

Mountpoint:

We create the mount point:

sudo mkdir -p /pgdata

Persistent mount via fstab:

We add the mount entry to /etc/fstab using the filesystem label rather than the device name. The device name (/dev/sdb) may change if disks are added or removed while the label remains stable:

echo 'LABEL=pgdata /pgdata xfs noatime,nodiratime 0 2' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /pgdata

We verify that the volume is mounted:

df -h /pgdata

PostgreSQL installation

We install PostgreSQL 18 from the official PGDG repository, which provides the latest PostgreSQL versions for Ubuntu:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt install -y postgresql-18

Cluster creation on /pgdata:

The Ubuntu packages create a default cluster under /var/lib/postgresql. This is not what we want. The data files and the WAL must both reside on the ZFS-backed volume, so that a single ZFS snapshot captures a consistent state of the database. If they were on different volumes, the snapshot would not be atomic.

We drop the default cluster and recreate it on /pgdata:

ahi@pgl:~$ sudo pg_dropcluster --stop 18 main
sudo install -d -o postgres -g postgres -m 700 /pgdata/18
sudo pg_createcluster -d /pgdata/18/main 18 main
sudo systemctl enable --now postgresql@18-main
Creating new PostgreSQL cluster 18/main ...
/usr/lib/postgresql/18/bin/initdb -D /pgdata/18/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /pgdata/18/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
             Ver Cluster Port Status Owner     Data directory  Log file
             18  main    5432 down   postgres  /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql@18-main.service → /usr/lib/systemd/system/postgresql@.service.

We verify that the cluster is online and located on the right volume:

ahi@pgl:~$ pg_lsclusters
Ver Cluster Port Status Owner     Data directory  Log file
18  main    5432 online <unknown> /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
ahi@pgl:~$ sudo -u postgres psql -c "SHOW data_directory;"
 data_directory
-----------------
 /pgdata/18/main
(1 row)

ahi@pgl:~$ sudo -u postgres psql -c "SELECT version();"
                                                             version
-------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
(1 row)

We can also confirm that the WAL directory lives inside the data directory, and therefore on the zvol:

ls -ld /pgdata/18/main/pg_wal

Creating a large database

We need a database large enough to make traditional backup and restore operations time-consuming. In the SQL Server part of this series, we used the StackOverflow database (about 207 GB). For PostgreSQL, we use pgbench, the benchmarking tool shipped with PostgreSQL.

We create the database and initialize it with a scale factor of 10000. This produces a database of approximately 146 GB, with 1 billion rows in the pgbench_accounts table:

sudo -u postgres createdb bench
sudo -u postgres pgbench -i -s 10000 --partitions=8 bench

A few minutes later:

We can monitor the data growth during the initialization:

watch -n 30 'df -h /pgdata'

A few minutes later:

On the Proxmox side:

After some time, the process completes:

vacuuming...

creating primary keys...
done in 1559.55 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 598.12 s, vacuum 675.53 s, primary keys 285.88 s).
ahi@pgl:~$

We check the database size:

We run a checkpoint before taking the snapshot. The recovery process starts replaying the WAL from the last checkpoint. By running it right before the snapshot, almost nothing needs to be replayed when the database starts after a restore:

sudo -u postgres psql -c "CHECKPOINT;"

Comparison with SQL Server:

On the SQL Server side, we had to run SUSPEND_FOR_SNAPSHOT_BACKUP and BACKUP WITH METADATA_ONLY. On the PostgreSQL side, none of that is needed.

The data files and the WAL are on the same zvol. An atomic ZFS snapshot therefore captures a state equivalent to a power loss, and PostgreSQL is designed to recover cleanly from that state through crash recovery: the WAL is replayed from the last checkpoint. This is documented and officially supported.

The snapshot is the backup. There is no .bkm file, no metadata backup.

Snapshot process flow

On the Proxmox side, we create the snapshot and protect it with a hold:

SNAP="sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)" 
zfs snapshot "$SNAP" 
zfs hold sqlsnap "$SNAP" 
echo "$SNAP"

The hold protects the snapshot from an accidental destruction, as we did in part 2 of this series. We note the exact snapshot name, it will be needed for the restore.

The database stays online during the whole operation. No I/O freeze is required.

We list the snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We drop the database then we restore the snapshot:

We run the snapshot restore procedure. On the VM, we stop the cluster and unmount the volume:

sudo systemctl stop postgresql@18-main
sudo umount /pgdata

On the Proxmox side, we want to restore our snapshot. We can list the available snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We roll back the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T165409

On the VM, we mount the volume and start the service:

sudo mount /pgdata
sudo systemctl start postgresql@18-main

We check a few elements in the logs:

sudo tail -30 /var/log/postgresql/postgresql-18-main.log

The service shutdown, then the restart:

UTC [232108] LOG:  database system is shut down
UTC [233968] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
UTC [233968] LOG:  listening on IPv4 address "0.0.0.0", port 5432
UTC [233968] LOG:  listening on IPv6 address "::", port 5432
UTC [233968] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

PostgreSQL detects that the database was not shut down properly and replays the WAL. This is the same crash recovery mechanism as in SQL Server. Finally, the database starts.

UTC [233974] LOG:  database system was not properly shut down; automatic recovery in progress
UTC [233974] LOG:  redo starts at 20/2CB76278
UTC [233974] LOG:  invalid record length at 20/2CB76380: expected at least 24, got 0
UTC [233974] LOG:  redo done at 20/2CB76348 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
UTC [233974] LOG:  checkpoint starting: end-of-recovery immediate wait
UTC [233972] LOG:  checkpoint complete: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.009 s, total=0.030 s; sync files=0, longest=0.000 s, average=0.005 s; distance=0 kB, estimate=0 kB; lsn=20/2CB76380, redo lsn=20/2CB76380
UTC [233968] LOG:  database system is ready to accept connections

We then verify that the database is available again:

sudo -u postgres psql -d bench -c "SELECT pg_size_pretty(pg_database_size('bench'));"

Consistency proof under load

The previous test was done on a quiesced database: we ran a CHECKPOINT right before the snapshot, and nothing was writing. The real question is: what happens if the snapshot is taken while the database is being written to?

This is where PostgreSQL differs the most from SQL Server. There is no SUSPEND_FOR_SNAPSHOT_BACKUP. We take the snapshot in the middle of the write activity and we let the WAL replay do the work.

We start the load. The built-in pgbench script runs a TPC-B-like transaction: three UPDATE statements on the accounts, tellers and branches tables and one INSERT into the history table:

sudo -u postgres pgbench -c 8 -j 4 -T 300 bench &

While the load is running, we take a snapshot on the Proxmox side:

zfs snapshot sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)

zfs hold sqlsnap sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551

No CHECKPOINT this time, no freeze. The database is actively writing while the snapshot is taken.

The state after some time under load:

We stop the service:

sudo systemctl stop postgresql@18-main 
sudo umount /pgdata

We restore the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551 

We mount the volume, start the service and check the logs:

sudo mount /pgdata 
sudo systemctl start postgresql@18-main

This time the log shows a real recovery:

2026-08-03 20:49:03.244 UTC [234405] LOG:  database system is shut down
2026-08-03 20:50:34.333 UTC [234620] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv6 address "::", port 5432
2026-08-03 20:50:34.335 UTC [234620] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2026-08-03 20:50:34.343 UTC [234626] LOG:  database system was interrupted; last known up at 2026-08-03 20:44:36 UTC
2026-08-03 20:50:34.381 UTC [234626] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-03 20:50:34.384 UTC [234626] LOG:  redo starts at 20/7A3618B0
2026-08-03 20:50:38.024 UTC [234626] LOG:  invalid record length at 20/8BC0A4F8: expected at least 24, got 0
2026-08-03 20:50:38.024 UTC [234626] LOG:  redo done at 20/8BC0A4D0 system usage: CPU: user: 0.71 s, system: 0.63 s, elapsed: 3.63 s
2026-08-03 20:50:38.028 UTC [234624] LOG:  checkpoint starting: end-of-recovery immediate wait
2026-08-03 20:50:53.936 UTC [234624] LOG:  checkpoint complete: wrote 105355 buffers (53.6%), wrote 5 SLRU buffers; 0 WAL file(s) added, 17 removed, 0 recycled; write=3.648 s, sync=12.232 s, total=15.911 s; sync files=189, longest=12.223 s, average=0.065 s; distance=287395 kB, estimate=287395 kB; lsn=20/8BC0A4F8, redo lsn=20/8BC0A4F8
2026-08-03 20:50:53.948 UTC [234620] LOG:  database system is ready to accept connections

Three differences compared to the first test:

• The “last known up at” timestamp (20:44:36) does not match a checkpoint we ran manually. It matches the last automatic checkpoint triggered during the load.

• The redo is not instantaneous anymore: 3.63 seconds, replaying about 280 MB of WAL (from LSN 20/7A3618B0 to 20/8BC0A4D0). All the write activity between the last checkpoint and the snapshot had to be replayed. The transactions committed before the snapshot are recovered, the ones that were in flight are rolled back.

• The end-of-recovery checkpoint then writes everything the redo rebuilt in memory: 105355 buffers, 53.6% of the buffer pool. The database is ready to accept connections about 19 seconds after the service start.

The crash recovery completed correctly and the database has been restored. We verify the TPC-B invariant. Each pgbench transaction applies the same delta to the accounts, tellers and branches tables in a single transaction. On a consistent database, the three sums must be equal:

Major drawbacks

• The snapshot covers the whole zvol. All the databases of the cluster are captured and restored together. There is no per-database restore, unlike the METADATA_ONLY approach on SQL Server which targets a single database.
• There is no backup history. SQL Server records the metadata backup in msdb. Here, the only trace is the snapshot itself on the ZFS side.
• Point-in-time recovery is not covered. The snapshot alone brings the database back to the moment it was taken. For PITR, WAL archiving would still be required on top of it.

Conclusion

• The snapshot backup and restore model of the SQL Server series applies to PostgreSQL (no I/O freeze, no metadata backup).
• One important rule: data files and WAL must reside on the same zvol so the snapshot is atomic.
• A 146 GB database was restored in a few seconds and in less than 20 seconds under active load, WAL replay included.

Thank you. Amine Haloui

L’article PostgreSQL Snapshot Backup and Restore with Proxmox ZFS (4/4) est apparu en premier sur dbi Blog.

After creating the Ticket, I connect to the instance and see that the msdb system database is in a Suspect State:

After taking a deep breath because I don’t see that every day, I begin my investigation by asking the state with the system view sys.databases first:

msdb is really in a suspect mode…

Now, to begin the analysis, I go to read the error log.:

The Error Log give me the error that I comments in SSMS query to read it better:

Error: 3314, Severity: 21, State: 1.
During undoing of a logged operation in database ‘msdb’, an error occurred at log record ID (821931:128397:5).
Typically, the specific failure is logged previously as an error in the Windows Event Log service. Restore the database or file from a backup, or repair the database.

If you have this error message, the best is to restore the msdb;
As the information of the backups is in the msdb, I cannot do a right-click to restore the database…  

I looked for the latest backup in the error log and find it:

But I also sse that the checkdb before each backup give 12 errors…

My question is at this point: Wasn’t there an alert saying that the database was corrupted?

As the users are blocked, I need to react quickly.
This is why, I do my restore and a CHECKDB with  REPAIR_ALLOW_DATA_LOSS:

RESTORE DATABASE msdb FROM DISK = 'xxx\msdb\msdb_fullbackup_xxxx20260801xxxx.BAK' WITH REPLACE;
GO
 ALTER DATABASE msdb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
GO
DBCC CHECKDB ('msdb', REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS, ALL_ERRORMSGS;
GO
ALTER DATABASE msdb SET MULTI_USER;

The commands completed successfully! YES!

The database msdb is online and working fine. SUPER!

To be sure, I go in the errorlog to see the restore and the checkdb with REPAIR_ALLOW_DATA_LOSS:

The restore was successful and in the checkDB with can see 12 errors and repaired 12 errors.

If I will investigate deeper, I will go the dump file and analyse it.

I finally managed to sort out and resolve the critical alert on monday morning.
Users can connect and read their tables again…
The customer knows why he comes to us for an SLA…

A good victory for a monday morning in our support SLA dbi-services!
See you soon for sharing again customer cases!

L’article SQL Server:  msdb is in Suspect State est apparu en premier sur dbi Blog.

We turn up on a Monday morning and there are alerts from the weekend saying

[MSSQL] – Instance Dark_Vador-Data(F:): Disk space in GB on server Death_Star is critical

Info: NOT OK – Free space: 20 MB out of 200 GB

The storage is almost full and:
– no projects were deployed over the weekend
– no exceptional data loads were scheduled
– nobody works at the weekend

A strange phenomenon to come across on a Monday morning….

On closer inspection, the findings are surprising; we can see that a database within the instance has been undergoing regular autogrowth for weeks…

Every day, there are several regular autogrowth on the data file for this database until the saturation of the disk.

Who is the culprit?

Not the business data, but one or more tables named xxxxLog or Historyxxx, or the combination of both HistoryLog!

PS: It’s an example but in the reality before the weekend, we will have Warnings and in the majority of the cases we react before the critical alert of course!

Data created to diagnose a problem, track an activity or maintain a history, which is then never cleaned up or has a default retention period that is too long and unsuitable for the context, is not always easy to detect.

On the other hand, it’s always easier to expand disk capacity without too much effort, and everything runs smoothly in an ideal world…

We, however, are Green SQL Server DBA; we’ll look deeper.
Our curiosity will lead us to challenge the business on these points.

The AUDIT

First, you need to know the autogrowth settings for these databases and how the data file growth.
Here is a simple script that shows the settings:

SELECT
    DB_NAME(database_id) AS DatabaseName,
    name AS LogicalFileName,
    type_desc AS FileType,
    physical_name AS PhysicalFileName,
    size / 128.0 AS CurrentSizeMB,
    CASE
        WHEN is_percent_growth = 1
            THEN CAST(growth AS VARCHAR(10)) + ' %'
        ELSE CAST(growth / 128 AS VARCHAR(10)) + ' MB'
    END AS AutoGrowth,
    max_size,
    CASE
        WHEN max_size = -1 THEN 'Unlimited'
        ELSE CAST(max_size / 128 AS VARCHAR(20)) + ' MB'
    END AS MaxSize
FROM sys.master_files where type_desc='ROWS' AND database_id > 4
ORDER BY
    DatabaseName,
    FileType,
    LogicalFileName;

Like for my precedent blog “Green SQL Server DBA Tips  #1 – Are unnecessary indexes cluttering up your database?”, I use the Dynamics database as example:

In my script, I exclude all system databases and the Log File to be concentrated to the Data File.

Now, I need to have the data file growth. This is through the default trace file and the event 92:

DECLARE @TraceFile NVARCHAR(500);

SELECT @TraceFile = path
FROM sys.traces
WHERE is_default = 1;

IF @TraceFile IS NULL
BEGIN
    RAISERROR('Default Trace is disabled.',16,1);
    RETURN;
END

;WITH GrowthEvents AS
(
    SELECT
        DB_NAME(DatabaseID) AS DatabaseName, FileName, CAST(StartTime AS DATE) AS GrowthDate,
        DATEPART(YEAR, StartTime) AS YearNum, DATEPART(WEEK, StartTime) AS WeekNum,
        DATEPART(MONTH, StartTime) AS MonthNum, IntegerData * 8.0 / 1024 AS GrowthMB
    FROM fn_trace_gettable(@TraceFile, DEFAULT)
    WHERE EventClass IN (92) -- 92 is data growth
)

SELECT DatabaseName,FileName,GrowthDate,COUNT(*) AS GrowthEvents,SUM(GrowthMB) AS TotalGrowthMB
FROM GrowthEvents GROUP BY DatabaseName,FileName,GrowthDate ORDER BY DatabaseName, GrowthDate

As I don’t have any growth on the dynamicsBC database, I switch to the M-Files database for my example:

We have here some daily growth…

Go to see if we have historical or log tables in M-Files with this simple script who give us  the table and the row count and MB:

SELECT s.name AS Schema_Name,t.name AS Table_Name,
    SUM(p.rows) AS Row_Count, CAST(SUM(a.total_pages) * 8.0 / 1024 AS DECIMAL(18,2)) AS SizeMB
FROM sys.tables t INNER JOIN sys.schemas s     ON t.schema_id = s.schema_id
INNER JOIN sys.indexes i ON t.object_id = i.object_id
INNER JOIN sys.partitions p ON i.object_id = p.object_id  AND i.index_id = p.index_id
INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id
WHERE t.name LIKE 'Histor%' OR t.name LIKE '%Log'
GROUP BY s.name, t.name ORDER BY SizeMB DESC;

The ANALISYS

In my case, we can see that we have 383550 lignes (19.73Mb) on the table named OBJECTTYPECHANGELOG for a M-Files database.

It’s only ~20MB here but in few months perhaps 20 GB or more…

As a Green SQL Server DBA, I will contact the owner of M-Files project otherwise don’t forget that dbi services has also the expertise in ECM tools like M-Files.

The IMPACT

Like in my precedent post, we will have the same impacts

The first impact is on DML INSERT operations.

With every insertion, the log or historical information is written to the table and update perhaps also index and at the end is never used by a query.

The second impact is the storage and the backup

If the log table represents ~20 GB of the database.

It’s 20 GB backed up unnecessarily, 20 GB restored unnecessarily and, above all, 20 GB stored unnecessarily and just for one environment.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 100 GB!

The third impact is on the maintenance plan

The following operations must also handle the log/historical tables:
– Rebuild Index
– Reorganise Index
– CheckDB
– Backups
– Restores

Each growth of log/historical tables extends the maintenance windows…

The ADVISE

You really need to be careful before deleting anything in these table.

Don’t Truncate the table because you see it!

First see with the vendor if a purge or a retention parameter exists through the application. 

In my case with M-Files and the table OBJECTTYPECHANGELOG, in the administration interface, you can configure the retention. Go to challenge your M-Files team on it!

Finally, for every large log or historical table, you need to ask yourself the following questions:
– Who uses it?
– How often?
– Is there a legal obligation?
– What retention period is required?
– Can it be archived?
– Can it be purged?

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will see a data disk full and says:

“How much GB need we to extend the disk?”

A Green SQL Server DBA sees a data disk full and says:

“Why is my disk full and what can I do?”

Just like adding a larger trunk doesn’t make a car more efficient, adding more GB doesn’t make a database healthier.

The greenest SQL Server DBA is not the one with the largest disks but it’s the one that keeps only data that continues to deliver value.

Think about it and begin now to see if you have cases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #2 – When logs and history records cause your database to growth and crash est apparu en premier sur dbi Blog.

Logical fragmentation was a problem on spinning disks: an ordered scan on a fragmented index broke the read-ahead into smaller I/Os and each jump cost a disk head movement. On SSD, NVMe and cloud storage, the penalty of non-sequential reads is marginal. And the “physical order” we used to restore so carefully is only the order of the pages inside the data file: below it, the file system, the SSD controller and the storage layer place the blocks wherever they want anyway.

Here is the mapping to remember

• Internal fragmentation = page density = fixed by automatic index compaction
• External fragmentation = page order = ignored by automatic index compaction (it can even increase and we will see it in the demo)

How does it work ?

Automatic index compaction is not a hidden maintenance job. It is an additional task performed by a background process that already exists: the PVS cleaner is a component of Accelerated Database Recovery (ADR).

Here is the principle

• The PVS cleaner periodically visits the pages that were recently modified (insert, update, delete) to remove obsolete row versions
• When automatic index compaction is enabled the cleaner also checks if the visited page has free space excluding the space reserved by the fill factor
• If so, it moves rows from the next page into the current page as long as they fit and repeats the operation on a few consecutive page pairs
• A page that becomes empty is deallocated

The result: the number of pages decreases, the page density increases, and the storage space, I/O, CPU and buffer pool consumption decrease.

The overhead is minimal because the process only considers recently modified pages, unlike a rebuild or a reorganize which process all the pages. Like a reorganize, the compaction acquires short-term exclusive page locks to move the rows. If a lock cannot be acquired immediately, the page is simply skipped and will be considered again later.

One command per database, no restart, no exclusive access. The compaction starts within minutes:

-- Enable the feature
ALTER DATABASE [SQL-DB-1] SET AUTOMATIC_INDEX_COMPACTION = ON;

-- Check if it's enabled
SELECT name, is_automatic_index_compaction_on FROM sys.databases;

One important point before you enable it

The compaction process only considers the pages modified after you enable the feature. If the page density of your indexes is already low, run a one-time reorganize or rebuild to fix the existing situation. From that point on, the automatic compaction keeps the indexes compact without any action on your side.

Demo

• We create a table with a clustered primary key and we insert 5 million rows
• We measure the baseline: page count, page density, fragmentation
• We delete 2 rows out of 3, scattered over the whole table, to simulate index bloat
• We measure again and we let the engine work

We create the table and we insert 50K rows:

CREATE TABLE dbo.Demo (
    Id      int IDENTITY CONSTRAINT PK_Demo PRIMARY KEY CLUSTERED,
    Payload char(400) NOT NULL DEFAULT REPLICATE('X', 400)
);

INSERT INTO dbo.Demo (Payload)
SELECT TOP (50000) REPLICATE('X', 400)
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

We measure the baseline:

SELECT index_level, page_count, avg_page_space_used_in_percent,
       avg_fragmentation_in_percent, record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Demo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

We measure the baseline:

SELECT index_level, page_count, avg_page_space_used_in_percent,
       avg_fragmentation_in_percent, record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Demo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

The internal fragmentation is low.

Here is what we have:

However to observe the automatic index compaction in action we need to create the following situation:

Here is the baseline:

We create the bloat

We delete 2 rows out of 3 scattered uniformly over the table. No page becomes empty. Every page keeps around one third of its rows. We change page density:

DELETE FROM dbo.Demo WHERE Id % 3 <> 0;

Without automatic index compaction, the expectation is simple: the page count stays at 277778 (deletes alone do not deallocate non-empty pages) and the density drops to around 31% (one third of 94.93%).

We measure again

We let the engine finish its job and we check again

The engine did the work in the background while the database stayed fully available.

This feature replaces the compaction part of your maintenance strategy not the whole strategy.

A summary:

• It does not update the statistics. A rebuild updates them the automatic compaction does not. If you rely on your rebuild jobs to refresh the statistics, keep a statistics update job.
• It does not reduce the logical fragmentation and it can even increase as we saw in the demo. For most workloads this has no measurable impact
• It does not recreate the free space reserved by the fill factor. Only a rebuild does. Workloads that need a fill factor below 100 to reduce page splits can still benefit from an occasional rebuild
• It does not shrink the data files. The used space decreases, the allocated size does not change
• It only applies to the leaf level of B-tree indexes in IN_ROW_DATA allocation units. Heaps, LOB data, row-overflow data, compressed columnstore rowgroups and memory-optimized tables are not concerned
• Indexes with page locks disabled (ALLOW_PAGE_LOCKS = OFF) are not eligible

Reference

Automatic Index Compaction – SQL Server | Microsoft Learn

Thank you. Amine Haloui

L’article SQL Server: Automatic index compaction in Azure (preview) – Part 2 est apparu en premier sur dbi Blog.

Microsoft recently released a new feature in public preview: automatic index compaction. The database engine takes care of compacting the indexes itself, in the background, while the data changes. No job, no maintenance window, no script to maintain.

Here is what we will cover in this blog post

• Which problems this feature addresses
• The requirements
• The difference between page density and index fragmentation (and why it matters)
• What the feature does not do

This is the first part of a series of two blog posts.

Which problems does it address?

Automatic index compaction is Microsoft’s answer to a routine operational task: index maintenance jobs.

Here are the problems with these jobs

• They are expensive in CPU and I/O. A rebuild reads and rewrites the entire index
• They take time to run, and maintenance windows keep shrinking
• They generate a lot of transaction log, which impacts log backups and Always On replication
• They can block applications. An offline rebuild locks the table and even an online rebuild needs a short exclusive lock at the beginning and at the end
• Someone has to set them up, monitor them and fix them when they fail

There is also a less obvious problem: for years, these jobs have been optimizing the wrong metric. We will come back to this point in a moment.

Requirements

The feature is currently in public preview and available on

• Azure SQL Database
• Azure SQL Managed Instance, with the Always-up-to-date update policy
• SQL database in Microsoft Fabric

It is not available on SQL Server on-premises, including SQL Server 2025.

There is nothing to install. The feature relies on a background process that already runs on these platforms (the ADR cleaner, more on this below), and Accelerated Database Recovery is always enabled on Azure SQL Database, Azure SQL Managed Instance and SQL database in Fabric.

Internal and external fragmentation

When we talk about index fragmentation, we are actually talking about two different phenomena. They are measured by two different columns of sys.dm_db_index_physical_stats, they have different causes, and above all they have very different costs.

Before we start: how is a data file organized ?

To understand fragmentation, we need one prerequisite: how SQL Server sees its data files.

• For any program, a file is an array of bytes provided by the operating system. Byte 0, byte 1, byte 2, and so on. Reading means asking for a range: an offset and a length

• SQL Server slices this array into pages of 8 KB. The page N is, by definition, the bytes N × 8192 to (N+1) × 8192 − 1. The page ID is not a label, it is an address in the file

• Everything below the file (NTFS clusters, SSD, SAN volumes) is translated and hidden by the storage stack.

Every time we say “physical order” in the rest of this post, we mean the order of the pages inside the data file not their placement on the hardware.

What does it look like?

Internal fragmentation: the pages are not full

Internal fragmentation means that the pages of an index contain free space. It is measured by avg_page_space_used_in_percent, also called page density.

Where does it come from?

• Deletes, especially scattered deletes: the rows disappear, the pages stay
• Page splits: when a page is full, an insert in the middle of the key range or an update that makes a row grow moves half of the rows to a newly allocated page. Both pages end up around 50% full
• Updates that shrink variable length columns
• Large rows that do not pack well: a 5000 bytes row means one row per page, and around 38% of every page is lost mechanically

Note that some free space is normal, and even intentional: the space reserved by the fill factor is internal fragmentation on purpose to absorb future inserts without page splits.

What does it look like?

CREATE DATABASE DemoFrag
GO

ALTER DATABASE DemoFrag SET RECOVERY SIMPLE
GO

USE DemoFrag
GO

CREATE TABLE dbo.SplitDemo (
    Id      int NOT NULL CONSTRAINT PK_SplitDemo PRIMARY KEY CLUSTERED,
    Payload varchar(8000) NOT NULL
);

-- 2 lines about 4000 bytes per page
INSERT INTO dbo.SplitDemo VALUES
(1, REPLICATE('A', 4000)), (2, REPLICATE('B', 4000));

My page is almost full:

Why does it cost ?

Because the same rows are spread over more pages than necessary and everything in SQL Server is done at the page level:

• The buffer pool caches pages, not rows. A page at 40% density wastes 60% of the memory it occupies
• Every read, logical or physical, handles more pages for the same data
• Backups are bigger, integrity checks are longer

This is the key point: internal fragmentation follows the data everywhere, including in RAM. It is a permanent tax on memory, I/O and CPU. However it’s not the case with external fragmentation.

External fragmentation: the offsets do not follow the logical order

At the leaf level of a B-tree, the pages are chained together in the order of the index keys with two pointers stored in every page header (m_nextPage, m_prevPage). This chain is the logical order and it cannot be wrong: it is the order of the keys by definition.

External fragmentation (also called logical fragmentation) means that following this chain no longer corresponds to reading the file sequentially. The pages are logically ordered, but their offsets in the file are not. It takes two forms:

• Holes: the pages of the index are not at consecutive offsets. Pages of other objects, or free pages, sit in between
• Disorder: the pages are at consecutive offsets, but chained in a different order

avg_fragmentation_in_percent counts both forms without distinction. The columns fragment_count and avg_fragment_size_in_pages are closer to the real cost: they measure the length of the contiguous runs.

Where does it come from?

• Several objects growing at the same time: the engine allocates the extents in the order the requests arrive, so the objects interleave in the file. This is the normal life of a database
• Page splits: the new page is allocated where free space is found, rarely next to the original page
• Page deallocations: massive deletes, and the automatic index compaction itself

What does it look like?

USE DemoFrag
GO

CREATE TABLE dbo.SplitDemo (
    Id      int NOT NULL CONSTRAINT PK_SplitDemo PRIMARY KEY CLUSTERED,
    Payload varchar(8000) NOT NULL
);

-- 2 lines about 4000 bytes per page : lines 1 and 2 on 1 page, 3-4 on next page

INSERT INTO dbo.SplitDemo VALUES
(1, REPLICATE('A', 4000)), (2, REPLICATE('B', 4000)),
(3, REPLICATE('C', 4000)), (4, REPLICATE('D', 4000))

-- Lines location (file:page:slot)?
SELECT Id, sys.fn_PhysLocFormatter(%%physloc%%) AS physical_location
FROM dbo.SplitDemo;

My first 2 rows are located on page 376 and the last 2 rows on page 378.

What does page 377 contain? I thought the pages were contiguous?

Since my table has a clustered index on Id it is stored as a B-tree. Can we see it?

SELECT allocated_page_page_id, page_type_desc, page_level, previous_page_page_id, next_page_page_id
FROM sys.dm_db_database_page_allocations(DB_ID(), OBJECT_ID('dbo.SplitDemo'),1, NULL, 'DETAILED');

What does it look like internally?

SELECT * FROM sys.dm_db_page_info (10, 1, 376, DEFAULT);

SELECT * FROM sys.dm_db_page_info (10, 1, 378, DEFAULT);

Page split to generate external fragmentation:

-- Split happens
UPDATE dbo.SplitDemo SET Payload = REPLICATE('B', 4100) WHERE Id = 2;

-- Check page density
SELECT index_level, page_count, avg_page_space_used_in_percent,
avg_fragmentation_in_percent, record_count,
ghost_record_count, version_ghost_record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.SplitDemo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

After the page split:

Line 1 is located at page 376

Line 2 is located at page 379

Line 3 is located at page 378

Line 4 is located at page 378

Why does it cost so little today?

There is exactly one mechanism through which external fragmentation ever costed something: the read-ahead. During a scan, the engine prefetches pages with large I/Os and an I/O is a contiguous range of the file: one offset, one length, up to 512 KB. When the index is contiguous one I/O brings back 64 pages. When it is not the same pages arrive in several smaller I/Os.

When do these smaller I/Os hurt ? Three conditions must be met at the same time:

• The data is cold: a page already in the buffer pool generates no physical I/O at all
• The operation is a large scan: a seek navigates the B-tree by page ID and does not care about the order
• Each additional I/O has a price

On SSD, NVMe and cloud storage, an additional I/O costs a few dozen microseconds and the physical placement below the file is arbitrary anyway (SSD controllers and SANs put the blocks wherever they want).

External fragmentation survived its own usefulness for another reason: it was a good symptom. It increases because of page splits, and page splits destroy the page density and inflate the transaction log.

Reference

Automatic Index Compaction – SQL Server | Microsoft Learn

Thank you. Amine Haloui

L’article SQL Server: Automatic index compaction in Azure (preview) – Part 1 est apparu en premier sur dbi Blog.