A customer recently asked me to disable FileTable/FileStream on their instance because the databases no longer use it, and to ensure that this is indeed the case.
The aim is to migrate to a new architecture without FileTable/FileStream in a second time.

Link to FileTable and FileStream if you need to know a bit more about these technologies

The environment is in a high-availability (HA) configuration using AlwaysOn.

As this isn’t something we do every day, here is a 9-step procedure to achieve this:

•    Check that there are no more FileTables

•    Disable non-transactional access

•    Check that non-transactional access is disabled

•    Check the columns in the tables

•    Identify FILESTREAM filegroups & check FILESTREAM files

•    Delete FILESTREAM containers & the FILESTREAM filegroup

•    Disable FILESTREAM on the SQL Server instance

•    Disable FILESTREAM on the server

•    Disable FILESTREAM on the other servers

Let’s get started!

Step 1: Checking the FileTables

Let’s start with the first, fairly simple step, which is to check that there are no more FileTables

Query used:

SELECT name,object_id,is_filetable FROM <database>.sys.tables WHERE is_filetable = 1;

Example:

In this example, it is clear that there is no longer a FileTable in the tables of the two databases checked here.

Step 2: Disabling non-transactional access

To ensure that no further data is sent to FileTable/FileStream, we disable non-transactional access:

Query used:

ALTER DATABASE <database> SET FILESTREAM ( NON_TRANSACTED_ACCESS = OFF );

Example:

The next step is, of course, to check carefully that it is disabled.

Step 3: Check that non-transactional access is disabled

Query used:

SELECT db_name(database_id),* FROM sys.database_filestream_options;

Example:

As can be seen in the example, all databases are set to OFF

Step 4: Checking the columns in the tables

This is a very important step, as it is the one I missed in my example by the customer…

In fact, in step 6, if there is still a column in a table that has a FileStream defined, it is not possible to delete the associated files and filegroups.

Query used:

SELECT * FROM <database>.sys.columns WHERE is_filestream = 1;
select * from sys.objects where object_id= xxxx

The first query returns the columns that contain FileStream data, and the second provides information about the table.

You will then find a structure of the following type:
CREATE TABLE [schema].[table](
 [ID] [uniqueidentifier] ROWGUIDCOL  NOT NULL,
…..
[FS] [varbinary](max) FILESTREAM  NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] FILESTREAM_ON [FileStreamFileGroup]

We can therefore see that there is indeed a FileStream that needs to be removed at column level, but also at table level with the FILESTREAM_ON [FileStreamFileGroup].

Initially, creating a new column with [FS2] [varbinary](max) NULL, copying the data and renaming it, seemed like a good idea but was not sufficient, as the table still had FILESTREAM_ON [FileStreamFileGroup].
To resolve this, you need to recreate a table, copy the data and add the keys, indexes and constraints.

Then check everything again before proceeding to the next steps.

Example:

Step 5:  Identify FILESTREAM filegroups & Check FILESTREAM files

In this step, the aim is to identify the FILESTREAM filegroups defined in the databases so that they can be deleted.

Query used:

SELECT name, type_desc FROM <database>.sys.filegroups WHERE type = 'FD';
SELECT name,physical_name,type_desc FROM <database>.sys.database_files WHERE type_desc = 'FILESTREAM';

Example:

As there are two databases in my example that have a FileStream, there are four lines.

Step 6: Delete the FILESTREAM containers and the FILESTREAM FILEGROUP

Once identified, delete them using the following command.
Query used:

ALTER DATABASE [Database]  REMOVE FILE [FileStreamFile]

Example:

Step 7: Disable FILESTREAM on the SQL Server instance

At this point, we disable FILESTREAM at instance level.

Query used:

EXEC sp_configure 'filestream access level';
GO
EXEC sp_configure 'show advanced options', 1;
RECONFIGURE;
EXEC sp_configure 'filestream access level',0;
GO
RECONFIGURE;
GO
EXEC sp_configure 'show advanced options', 0;
RECONFIGURE;
GO

Example:

As can be seen from the example, the `config_value` is indeed 0, but the `run_value` has changed to 1.

In such cases, as always, we’ll check what the documentation says:

We are in T-SQL access only, it’s not really desactivated… Need a restart!

At this stage, we could also carry this out on the secondary node(s) of our AlwaysOn high availability cluster, but it is preferable to do it first on a single node and verify that it works.

Now it is time to disable Filestream on the SQL Server Engine service.

Step 8: Disabling FileStream on the server

For this step, we have already performed a failover of the AAGs to a synchronous secondary node.

We launch SQL Server Configuration Manager to disable FileStream in the Engine properties by unticking the box, as shown in the example above:

Then restart SQL Server and check that FileStream is disabled on the instance:

Step 9: Disabling FileStream on the other servers

At this point, you will just need to repeat steps 7 and 8 on the other nodes in your AlwaysOn high-availability cluster.

Conclusion

The steps are not complex in themselves, but where you can lose time is in the step 4 if you have to recreate a table without the FileStream and copy all data from this table. I was happy to have only one with not so much rows…

All the steps up to the restart of the SQL Server Engine service in step 8 are carried out online without any service interruption or failover.

I hope these steps will help anyone facing a similar issue!

L’article SQL Server: Disabling FileTable/FileStream on an AlwaysOn high-availability instance est apparu en premier sur dbi Blog.

But what happens under heavy pressure, when hundreds of sessions make that call at the same moment and the model takes its time to answer? I ran this scenario on SQL Server 2025 and on Azure SQL Database, with the same procedure and the same load script. The two platforms do not react the same way at all.

The documentation of the procedure describes a cap at 10% of the worker threads. That is not what I measured. Azure SQL Database does cap the calls in my tests, but at 15% of its workers: 22 calls at 2 vCores, 45 at 4 vCores, 30 on a Standard S3. SQL Server 2025 has no cap at all, and the rest of this post shows what happens without it.

The lab

The on-premises setup has three parts, all on one virtual machine (Windows Server 2022, 2 vCPUs): a load generator, a SQL Server 2025 Standard instance, and a small HTTPS endpoint that answers after the delay given in the URL.

The call is wrapped in a procedure, so that every session runs exactly the same thing:

CREATE OR ALTER PROCEDURE dbo.CallSlowAi @delay int, @timeout smallint = 230
AS
BEGIN
    DECLARE @url nvarchar(400) = CONCAT(N'https://<YOUR_VM>:8443/?delay=', @delay);
    DECLARE @ret int, @resp nvarchar(max);
    EXEC @ret = sp_invoke_external_rest_endpoint @url = @url, @method = 'POST',
         @payload = N'{"input":"hello"}', @timeout = @timeout, @response = @resp OUTPUT;
    RETURN @ret;
END

The load generator is a PowerShell function that opens N connections without pooling, sends EXEC dbo.CallSlowAi @delay = 30 on all of them at the same moment, and records for each session the elapsed time and the error number. One process, so the client itself does not slow the VM down.

Before measuring SQL Server, I called the endpoint directly with 200 parallel requests: they all came back in 30.6 seconds. The endpoint is not the bottleneck.

On-premises: no limit, until the workers run out

A worker thread is what SQL Server uses to execute a request: one request running, at least one worker busy. The instance has a fixed pool of them, set by max worker threads; with the default setting on this 2-vCPU machine, that is 512. When they are all taken, new work waits with the THREADPOOL wait type, and that includes logging in, because a login needs a worker too (see my previous blog about this wait type).

Three words matter here, and their numbers should not be added together. A session is a connection. When it runs a request, the request is split into one or more tasks, and each task needs a worker thread to run. A simple call like ours is one task, so one worker; a parallel query would take several. The worker counts I quote also include SQL Server’s own background tasks: about 45 workers on this idle instance. A wait type, on the other hand, is counted per task, whoever owns it.

A session waiting for the endpoint reports the HTTP_EXTERNAL_CONNECTION wait type and keeps its worker thread for the whole wait: every waiting call added one worker to that baseline.

With 300 sessions, SQL Server sent everything at once. I counted 300 TCP connections from sqlservr.exe to port 8443, and all calls returned together:

ErrorNumber : 0
Sessions    : 300
MinSec      : 30.2
MaxSec      : 34.5

With 600 sessions, the instance has only 512 worker threads to give. 510 connections were open at the same time, and the calls finished in two groups:

Finished after (s)   Sessions
30                   193
40                   316
50                     1
60                    90

The first 509 calls left immediately, one per available worker. The last 91 did not wait for the endpoint: they waited for a worker. No error was raised at any point, and nothing held the calls back.

What the rest of the instance sees

I measured the saturation itself during an earlier run of the same 600 sessions, with HTTP/2 still enabled on the endpoint (see below). From the dedicated admin connection:

active_workers  queued_tasks
556             154

-------------------------------------

wait_type                  n
HTTP_EXTERNAL_CONNECTION   513
THREADPOOL                 154

At that moment an ordinary sqlcmd connection failed:

Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login timeout expired.
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Unable to complete login process due to delay in prelogin response.

The admin connection kept answering, and the instance recovered by itself once the calls completed. KILL also works at once on a waiting session (error 596 on the client side).

HTTP/2 changes the shape, not the outcome

HTTP.sys, which hosts my test endpoint, offers HTTP/2 by default. In that mode SQL Server opened 6 TCP connections instead of one per call, and only 100 calls were in flight per host name. The same 300 sessions then finished in three waves:

Finished after (s)   Sessions
30                   100
60                   100
90                   100

The queued sessions show the same HTTP_EXTERNAL_CONNECTION wait and hold a worker while they queue, so the pool empties just as well: the 556 workers above were measured in this mode. The queue has a second effect. @timeout keeps running while a call waits its turn. With a 30-second endpoint and @timeout = 45, the first 100 calls succeeded and the 100 queued ones failed after 45.5 seconds with error 31629, The external rest endpoint execution exceeded the timeout value, although the endpoint was answering normally.

Turning HTTP/2 off on the endpoint removes that queue. I disabled it in HTTP.sys (EnableHttp2Tls and EnableHttp2Cleartext set to 0, then a reboot) and replayed the same load:

The waves are gone and so are the timeouts caused by the queue. It is not a fix for the instance, though. Each call still holds a worker, the 600-session run at the top of this section was made in HTTP/1.1, and the endpoint now receives every call at once. With a real AI provider you also do not choose the protocol: the server does.

I did not capture the HTTP/2 negotiation, so I cannot say which side sets the figure of 100. What the two modes show together is that SQL Server applies no cap of its own.

But the documentation says 10%?

The page of the procedure is marked as applying to SQL Server 2025, Azure SQL Database, Azure SQL Managed Instance and SQL database in Microsoft Fabric. Its throttling section reads:

The number of concurrent connections to external endpoints done via sp_invoke_external_rest_endpoint are capped to 10% of worker threads, with a maximum of 150 workers.

It adds that a call made beyond the limit raises error 10928. With 512 worker threads, I expected about 51 calls to pass and the others to fail. I got neither the cap nor the error, on RTM as on CU9.

The next sentence of that section gives a hint: it speaks of a single database and of an elastic pool, two notions that only exist in Azure SQL Database. So I ran the same test there.

Azure SQL Database: 22 calls, then an error

Azure SQL Database only calls endpoints from a closed list of Azure services, so the slow endpoint becomes an Azure Function. The database is a single Azure SQL Database in Switzerland North. I started on the vCore model, General Purpose serverless on standard-series (Gen5) hardware with 2 vCores maximum, then scaled the same database to 4 vCores, and finally to Standard S3 in the DTU model. The function runs on the Consumption plan, Node.js 22, in the same region. The procedure is the same, with the URL of the function, and the load script only changes its connection string.

Before any load, the governance view of the database already shows a dedicated limit:

SELECT primary_group_max_workers, primary_group_max_outbound_connection_workers
FROM sys.dm_user_db_resource_governance;

primary_group_max_workers  primary_group_max_outbound_connection_workers
150                        22

I stated my bet before running the load: 22 successes, 78 errors. Result for 100 sessions calling a 30-second endpoint:

ErrorNumber  Sessions  Finished after (s)
0            22        30
10928        78        0

Resource ID : 4. The outbound connection worker limit for the database is 22 and has been reached.

The 78 rejected sessions got their answer immediately, which leaves the database 128 of its 150 workers for everything else, whatever the endpoint does.

Two remarks on this test. The ratio is 15%, not the documented 10%. At 2 vCores the database has 150 workers and a limit of 22. I scaled it to 4 vCores: 300 workers, a limit of 45, and the same load gave 45 successes and 55 errors. In the DTU model, a Standard S3 shows 200 workers and a limit of 30. This is one database, in one region, read on three service objectives and load-tested on two. The ratio could differ on other tiers, hardware generations or Hyperscale, and Microsoft may change it: treat 15% as what I observed, not as a rule. On the endpoint side I needed to do a small adjustment as well: an Azure Function on the Consumption plan served only 100 concurrent requests per instance until I raised maxConcurrentRequests in host.json. A direct test of 200 parallel requests took 62.8 seconds before the change and 30.4 after.

The same limit on-premises, with Resource Governor

SQL Server 2025 exposes nothing comparable. On CU9, sys.dm_user_db_resource_governance does not exist, no system column deals with outbound connections, and the only related configuration option is the on/off switch external rest endpoint enabled.

What SQL Server does have is Resource Governor, and a workload group can limit its number of concurrent requests. I created a group with the Azure value and sent the sessions of the load generator into it through their application name:

USE master;
GO
CREATE WORKLOAD GROUP AiCalls WITH (GROUP_MAX_REQUESTS = 22) USING [default];
GO
CREATE FUNCTION dbo.RgClassifier() RETURNS sysname WITH SCHEMABINDING
AS
BEGIN
    RETURN CASE WHEN APP_NAME() = N'AiLoad' THEN N'AiCalls' ELSE N'default' END;
END
GO
ALTER RESOURCE GOVERNOR WITH (CLASSIFIER_FUNCTION = dbo.RgClassifier);
ALTER RESOURCE GOVERNOR RECONFIGURE;

Same load as on Azure, 100 sessions calling a 30-second endpoint:

ErrorNumber  Sessions
0            100

Finished after (s)   Sessions
30                   22
60                   22
90                   22
120                  22
150                  12

The limit is applied to the unit: never more than 22 calls reach the endpoint. But there is no error. Every session ends up being served, and the last ones wait two minutes for it.

Where Resource Governor stops

During the second wave of that test, the group, the waits and the workers looked like this:

The 56 queued sessions wait on RESMGR_THROTTLED, and each of them has a worker. The instance counts 123 active workers against 45 at rest: 78 more, exactly the 22 calls in flight plus the 56 sessions in the queue.

Resource Governor protects the service being called. It does not protect the instance. With 600 sessions classified in that group, 22 calls would be in flight and the 578 others would still be queued, each holding a worker, against a pool of 512. I did not run that last case; it follows from the two measurements above.

What to take from this

On SQL Server 2025, every session waiting for an external endpoint holds a worker thread, and the engine sets no limit on how many can wait at once. Azure SQL Database does have such a cap, at 15% of its workers in my tests rather than the documented 10%, and a call beyond it is refused immediately. On-premises, a slow AI service plus a concurrent application is enough to exhaust the worker pool and to keep new connections out, with no blocking chain to look for and an error message, when there is one, that points at the endpoint.

The difference is also one of scope. In Azure SQL Database, the limit belongs to the database: a flood of slow calls is refused at the door of that database, and the 128 workers left keep it running. In SQL Server, the worker pool belongs to the instance. Every database on it shares the same 512 workers, so one application calling a slow AI service can lock out the users of all the others.

Resource Governor can cap the number of calls sent to the service, in Standard Edition too, but it queues the surplus instead of refusing it, and a queued session costs a worker. If you want the Azure behavior, the refusal has to come from the code that makes the call, or from the application in front of it. Make sure you can reach the dedicated admin connection before you need it: it works locally out of the box (sqlcmd -A), but from another machine it requires the remote admin connections option. It was the only way in during the saturation, and KILL on the waiting sessions frees their workers at once.

L’article Calling AI from SQL Server 2025 can lock everyone out of your on-prem instance est apparu en premier sur dbi Blog.

This part is about putting it back: translating the views, rebuilding the keys and indexes under a Standard Edition licence that makes every build offline, and compressing 3.2 TB of heaps that turned out to be mostly empty space.

The logbook resumes at day 5.

Day 5. Four thousand views to convert

4,658 of the 4,784 views live in a single schema, and nobody needed all of them. We went through the list with the client first and kept the ones that actually matter to the business, which brought the scope down to about 710. SSMA converted around 600 of those on its own.

SSMA fails mostly on CONNECT BY, START WITH and SYS_CONNECT_BY_PATH. For some easy cases, DECODE to CASE and NVL handles it well, for the record. For more complex use case, we needed to find an automatic solution process. So first, we discussed with the client to know which view were critical and needed to be migrated. Then, with this list, we leveraged the power of AI to help us converting the remaining views based on the following workflow:

The workflow

SSMA converts what it can. The failures get extracted from dba_views into files. Then, and this is the step that changes the economics, they get classified by pattern in the Oracle source rather than one at a time by error message.

Translation itself was done in batches of one family at a time, outside production, on extracted files, with an AI assistant and a context file describing the type mapping decisions, the column types of the referenced tables and the target version. Then human review, tests against Oracle and finally, deployment.

The catalogue of families is the real deliverable. Eight of them closed the great majority of the cases:

And a word on the AI, since that is the part people ask about. No data ever left the environment: only the view definitions were processed, and every test against the live database was run by hand so we kept control of the process. We could have added MCP connectors and let a local model run the tests itself, but with the number of days allocated to the project and the small number of views left to fix, doing it manually was simpler. Around 85 percent of the views wanted by the client had already come through SSMA.

Day 6. Putting the indexes back and making the database smaller

Recovering what I had thrown away

The primary keys I dropped to load into heaps had left no trace on the SQL Server side. Nothing in sys.indexes, no disabled definition, nothing.

What Standard Edition costs

Index builds are offline and single-threaded. The table is locked for the duration, metadata reads queue behind it, but we managed to run 4 builds in parallel. SORT_IN_TEMPDB = OFF for the clustered builds. The sort covers the entire table, and 36 GB of tempdb was never going to hold a 900 GB sort.

Meanwhile percent_complete sits at 0 for the whole build, with the serene confidence of something that has never been asked to account for itself. It works for BACKUP DATABASE. It does not work here. Real progress lives in sys.dm_exec_query_profiles, operator by operator :

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SELECT physical_operator_name, row_count, estimate_row_count,
       CAST(row_count*100.0/NULLIF(estimate_row_count,0) AS decimal(5,1)) AS pct
FROM sys.dm_exec_query_profiles WITH (NOLOCK)
WHERE session_id = <spid of the build> 
ORDER BY node_id;

Those are not three parallel counters. They are three sequential phases of one pipeline, and only one of them moves at a time. The scan reads the source table, and while it runs its percentage is the only honest progress figure available. The sort is a blocking operator: it cannot emit a single row until it has consumed all 31.9 million, so it stays at zero through the whole scan and then through the whole sort. That is the blind spot. Once the scan reaches 100 percent you get a long stretch where all three counters report nothing while the server works flat out, and if the sort spills to disk that stretch can outlast the other two phases combined. Index Insert starts moving only when the sort begins handing rows over, and it is the first number that means the end is actually in sight.

Why compress, and what it returned

On a read-only database, PAGE compression has no recurring write cost. You keep the read benefit and pay nothing back. The argument is rarely this clean.

Compression trades CPU for I/O, and on an archive that trade is one-sided. These pages are read rarely, and when they are, nothing heavy is competing for the processor, so the decompression cost lands on a machine with nothing better to do. What you get back is density: more rows per page, which means fewer reads to find the one row somebody asked for and more of the table sitting in memory at once.

The data was unusually well suited to it, for a reason worth spelling out. These tables carry roughly 200 numeric columns, mostly null, plus five nvarchar(200) aggregation labels that repeat endlessly. Here is the part that surprises people: a fixed-length numeric column that is NULL still occupies its full width in SQL Server. “Mostly null” saves nothing before compression. It is precisely why the heap was that large, and precisely what the page dictionary destroys.

The allocated space per row tells the story:

Three tables hold 2.7 TB of the 3.2 TB total. At around 4 KB of allocated space per row against an 8 KB page, the top two fit around two rows per page and waste most of what is left.

Then the first build, on 505,819,544 rows with a five-column primary key:

ALTER TABLE <TABLE6>
ADD CONSTRAINT PK_TABLE6 PRIMARY KEY CLUSTERED
    (<COLUMNS_NAME>)
WITH (DATA_COMPRESSION = PAGE, SORT_IN_TEMPDB = OFF);

117 GB to 30.7 GB. A ratio of 3.81.

Finally, as you can see above, more than 2.5 T have been reclaimed (datafiles were full with the heaps). Compression has made huge benefits and that means less data on disks, which means fewer pages to take in the buffer (less IO).

The script that puts everything back

Across every schema that is two cursors, one rebuilding the disabled indexes with compression and one re-enabling the foreign keys. It is driven by the same #Schemas table as the disable script in part 1:

SET NOCOUNT ON;
DECLARE @Execute bit = 0;   -- 0 = preview ; 1 = execute
  
DECLARE @cmd nvarchar(max);
DECLARE @n   int = 0;
  
DECLARE reb_cur CURSOR LOCAL FAST_FORWARD FOR
    SELECT N'ALTER INDEX ' + QUOTENAME(i.name)
         + N' ON ' + QUOTENAME(s.name) + N'.' + QUOTENAME(t.name)
         + N' REBUILD WITH (DATA_COMPRESSION = PAGE);'
    FROM   sys.indexes  i
    JOIN   sys.tables   t ON i.object_id = t.object_id
    JOIN   sys.schemas  s ON t.schema_id = s.schema_id
    WHERE  s.name IN (SELECT schema_name FROM #Schemas)
      AND  i.is_disabled = 1          
    ORDER BY s.name, t.name, i.name;
  
OPEN reb_cur;
FETCH NEXT FROM reb_cur INTO @cmd;
WHILE @@FETCH_STATUS = 0
BEGIN
    SET @n += 1;
    PRINT @cmd;
    IF @Execute = 1
    BEGIN
        BEGIN TRY EXEC sys.sp_executesql @cmd; END TRY
        BEGIN CATCH PRINT N'   -- FAILED: ' + ERROR_MESSAGE(); END CATCH
    END
    FETCH NEXT FROM reb_cur INTO @cmd;
END
CLOSE reb_cur;
DEALLOCATE reb_cur;
  
PRINT N'---';
PRINT CONVERT(nvarchar(10), @n) + N' index '
    + CASE WHEN @Execute = 1 THEN N'rebuilt.' ELSE N'to be rebuilt (preview).' END;
GO
  
  
SET NOCOUNT ON;
DECLARE @Execute bit = 1;   -- 0 = preview ; 1 = executer
  
DECLARE @cmd nvarchar(max);
DECLARE @n   int = 0;
  
DECLARE fkc_cur CURSOR LOCAL FAST_FORWARD FOR
    SELECT N'ALTER TABLE ' + QUOTENAME(s.name) + N'.' + QUOTENAME(t.name)
         + N' WITH NOCHECK CHECK CONSTRAINT ' + QUOTENAME(fk.name) + N';'
    FROM   sys.foreign_keys fk
    JOIN   sys.tables  t ON fk.parent_object_id = t.object_id
    JOIN   sys.schemas s ON t.schema_id = s.schema_id
    WHERE  s.name IN (SELECT schema_name FROM #Schemas);
  
OPEN fkc_cur;
FETCH NEXT FROM fkc_cur INTO @cmd;
WHILE @@FETCH_STATUS = 0
BEGIN
    SET @n += 1;
    PRINT @cmd;
    IF @Execute = 1
    BEGIN
        BEGIN TRY EXEC sys.sp_executesql @cmd; END TRY
        BEGIN CATCH PRINT N'   -- FAILED : ' + ERROR_MESSAGE(); END CATCH
    END
    FETCH NEXT FROM fkc_cur INTO @cmd;
END
CLOSE fkc_cur;
DEALLOCATE fkc_cur;
  
PRINT N'---';
PRINT CONVERT(nvarchar(10), @n) + N' FK '
    + CASE WHEN @Execute = 1 THEN N'activated and validated.' ELSE N'to be activated (preview).' END;
GO

The foreign keys, which turned out to be free

All 6,072 foreign keys were already back, active and marked not trusted, because the loading script re-enabled them with NOCHECK. Revalidating them with WITH CHECK would mean full joins against the largest tables in the database.

On a read-only archive whose source integrity was enforced by Oracle for fifteen years, that buys one thing: the optimizer regains the right to eliminate redundant joins. Worth having on a schema of deeply nested views, not worth days of scanning here. They stayed not trusted.

Day 7. Testing 600 views

The principle is simple: run every view, record what happened, and never lose the record. Each one gets a SELECT TOP (100) * inside a TRY/CATCH, so a view that fails is logged rather than fatal, and the outcome lands in a table with its duration in milliseconds. The cursor only picks up views that are not already recorded, which means an interrupted run resumes where it stopped instead of starting over, and the duration column is what you sort on once it is done:

SET NOCOUNT ON;
 
IF OBJECT_ID('dbo.ViewTestResults') IS NULL
    CREATE TABLE dbo.ViewTestResults (
        schema_name  sysname,
        view_name    sysname,
        statut       varchar(20),
        duree_ms     int,
        message      nvarchar(2048),
        tested_at    datetime2 DEFAULT SYSDATETIME(),
        CONSTRAINT PK_ViewTestResults PRIMARY KEY (schema_name, view_name)
    );
 
DECLARE @sch sysname, @vue sysname, @sql nvarchar(max), @t0 datetime2, @ms int, @i int = 0, @n int;
 
SELECT @n = COUNT(*)
FROM   sys.views v JOIN sys.schemas s ON v.schema_id = s.schema_id
WHERE  s.name = 'SCDAT'
  AND  NOT EXISTS (SELECT 1 FROM dbo.ViewTestResults r
                   WHERE r.schema_name = s.name AND r.view_name = v.name);
 
RAISERROR('=== %d view remaining ===', 0, 1, @n) WITH NOWAIT;
 
DECLARE c CURSOR FAST_FORWARD FOR
  SELECT s.name, v.name
  FROM   sys.views v JOIN sys.schemas s ON v.schema_id = s.schema_id
  WHERE  s.name = 'SCDAT'
    AND  NOT EXISTS (SELECT 1 FROM dbo.ViewTestResults r
                     WHERE r.schema_name = s.name AND r.view_name = v.name)
  ORDER  BY v.name;
 
OPEN c;
FETCH NEXT FROM c INTO @sch, @vue;
WHILE @@FETCH_STATUS = 0
BEGIN
    SET @i += 1;
    RAISERROR('[%d/%d] %s.%s', 0, 1, @i, @n, @sch, @vue) WITH NOWAIT;
 
    SET @t0 = SYSDATETIME();
    BEGIN TRY
        SET @sql = N'SELECT TOP (100) * FROM ' + QUOTENAME(@sch) + N'.' + QUOTENAME(@vue) + N';';
        EXEC sp_executesql @sql;
        SET @ms = DATEDIFF(MILLISECOND, @t0, SYSDATETIME());
        INSERT dbo.ViewTestResults (schema_name, view_name, statut, duree_ms, message)
        VALUES (@sch, @vue, 'OK', @ms, NULL);
    END TRY
    BEGIN CATCH
        SET @ms = DATEDIFF(MILLISECOND, @t0, SYSDATETIME());
        INSERT dbo.ViewTestResults (schema_name, view_name, statut, duree_ms, message)
        VALUES (@sch, @vue, 'FAILED', @ms, ERROR_MESSAGE());
        RAISERROR('    -> FAILED: %s', 0, 1, @vue) WITH NOWAIT;
    END CATCH
 
    FETCH NEXT FROM c INTO @sch, @vue;
END
CLOSE c; DEALLOCATE c;
 
RAISERROR('=== Done ===', 0, 1) WITH NOWAIT;
 
-- summary
SELECT statut, COUNT(*) AS nb, SUM(duree_ms) AS duree_totale_ms
FROM dbo.ViewTestResults GROUP BY statut;
 
-- failed
SELECT schema_name, view_name, duree_ms, message
FROM dbo.ViewTestResults WHERE statut='FAILED' ORDER BY view_name;
 
-- all
SELECT schema_name, view_name, statut, duree_ms
FROM dbo.ViewTestResults ORDER BY duree_ms DESC;

The result looked excellent: 604 views OK, zero errors, against real data. Six skipped because they are not implemented to be run without an explicit filter added. SELECT TOP 100 * without a filter is not a representative test but still confirms that definition and joins are working fine. So: test with the filter the users will actually write. A view that only misbehaves when queried in a way nobody queries it is not a broken view, but you cannot know that from a green tick.

Day 8. Closing the hatch

Database set to read-only, final backup taken and user’s specific queries tested with success, the story now comes to a happy end.

What I would tell the next person

An archive migration is not a cheap application migration. It is a different decision set, and read-only is the input that drives most of it. 25,000 triggers are pointless, PAGE compression free, and untrusted foreign keys perfectly acceptable. It even tells you how to index: broadly and in advance, because nobody can tell you which row an auditor will want in three years. Before anything else, ask what the target will be used for. The answer rewrites the plan before the first byte moves.

Measure, and distrust anything that presents itself as progress. The SSMA counter refreshes per batch and sits still for minutes at a time. percent_complete stays at zero for an entire index build. A green tick on SELECT TOP 100 confirms that the joins resolve and nothing beyond that. In every one of those cases the real number was sitting in a DMV.

On views, the conversion rate is the least useful number in the report. What made four thousand views a finite piece of work was not the translation but the two steps around it: agreeing the real scope with the client first, then classifying what remained by pattern in the Oracle source rather than one error message at a time. An AI assistant translates a family of twenty views very well. It validates nothing, and the test against the source stays yours.

Compression on an archive is the rare decision with nothing to weigh against it. 117 GB down to 30.7 GB on the first table, more than 2.5 TB reclaimed across the set, and not a single write to pay it back with. If your heap is mostly null numeric columns, it is bigger than you think, and it will compress better than you expect.

L’article Archiving 5 TB from Oracle into SQL Server: views, indexes and compression (part 2) est apparu en premier sur dbi Blog.

Reference: https://support.microsoft.com/en-us/servicing/sql/sql-server-2025/cumulative-update/kb5084896-cu5

I built a lab on CU5 and on later builds up to CU9 and I spent some time measuring. Most of the documentation is confirmed. The sentence that says what the option actually changes, is not: on my lab, under the conditions described below I could not reproduce it. This post is what I measured with the protocol so that you can run it yourself and tell me if you get something different.

Why would released locks still cost memory?

A lock is not one structure in memory, it is two. The lock block describes the locked resource: this row, this page, this table. The lock owner block describes who holds it and in which mode. Three sessions reading the same row give one lock block and three lock owner blocks. sys.dm_tran_locks returns one row per owner block: the resource_* columns describe the lock block, the request_* columns the owner block and lock_owner_address is literally its address in memory.

The documentation gives 96 bytes per lock: 64 bytes for the lock block, plus 32 bytes per owner block. On my lab, a row lock costs about 195 bytes all included, the clerk’s pages divided by the number of locks held: twice the documented figure. Part of the gap is known. The 64 bytes are the 32-bit size of the lock block: on 64-bit, Kalen Delaney gives 128 bytes. That leaves about 67 bytes for the owner block, the share of the intent lock on the page and the overhead of the object store. Keep 200 bytes in mind. Ten million row locks are 2 GB.

These structures are not database pages. They are stolen memory taken from a pool owned by the lock manager. This pool is visible as the OBJECTSTORE_LOCK_MANAGER memory clerk in sys.dm_os_memory_clerks (one clerk per NUMA node, named Lock Manager : Node 0…) and as the Lock Memory (KB) performance counter. The locks documentation says 2500 lock structures. On my lab, Lock Blocks Allocated reports 3050 right after a restart, on SQL Server 2022 as on 2025.

Reference : https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-the-locks-server-configuration-option?view=sql-server-ver17

Here is what I have :

SELECT
    cntr_value AS lock_blocks_allocated
FROM sys.dm_os_performance_counters
WHERE object_name LIKE N'%:Memory Manager%'
  AND counter_name = N'Lock Blocks Allocated';

Now the important part. When a transaction commits or rolls back, its locks are released. The lock structures are not returned to SQLOS. They stay in the pool, ready to be reused by the next lock request. This is a cache and the reason is simple: allocating memory for every lock acquisition would be expensive and lock acquisition is on the hot path of every query.

The numbers to remember

Four numbers. The documentation expresses the percentages against the “SQLOS committed memory”. On my lab, they behave as percentages of committed_target_kb in sys.dm_os_sys_info, the Target Server Memory (KB) counter: what SQLOS is allowed to use, not what it has used so far. Test 1 below shows why I say so.

Two remarks. Lock escalation exists precisely to keep lock memory small: 5000 row locks cost about 1 MB, one table lock costs 200 bytes. And the 60 % limit is not a cache limit: it is the maximum memory the lock manager may use, cache included.

What was the problem before CU5?

The cache could grow up to the 60 % limit. It was given back slowly, by a background task and that task could be slow enough for the cache to look permanent.

The typical trigger is a workload that acquires a very large number of locks at once. The documentation mentions large concurrent query workloads with lock escalation disabled at the instance level (trace flag 1211). But you do not need a trace flag for that. ALTER TABLE … SET (LOCK_ESCALATION = DISABLE) on a big table, a reporting query in REPEATABLE READ or SERIALIZABLE scanning millions of rows, a purge that could not escalate because another session held an incompatible lock on the table: any of these can push lock memory to several GB.

After the transaction ends, the lock structures stay in the cache. The buffer pool, the plan cache and every other cache have to live with what is left. Page life expectancy drops, physical reads increase, and nothing in the current workload explains it, because the workload that caused it is over.

The symptom is easy to recognize once you know it. An idle instance, sys.dm_tran_locks almost empty and OBJECTSTORE_LOCK_MANAGER reporting GB. On an instance with a 128 GB target, the cache can hold up to about 77 GB.

What the documentation says CU5 changes

The lock manager cache now has a ceiling and you can configure it:

EXECUTE sp_configure 'show advanced options', 1;
RECONFIGURE;

EXECUTE sp_configure 'max lock manager cache memory (%)', 20;
RECONFIGURE;

The server must be restarted before the setting can take effect. The Server configuration options page lists it as A, RR (advanced, restart required): https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/server-configuration-options-sql-server?view=sql-server-ver17

What the option limits is the cache, not the locks.

As stated in the documentation:

Lock manager memory can still grow up to 60 percent of the SQLOS committed memory if required by the workload. However, when locks are released, memory is freed rather than cached if the lock manager cache already reached its configured limit.

Reference : https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/max-lock-manager-cache-memory-configuration-option?view=sql-server-ver17

It says two things. A transaction may still take up to 60 % of the memory in locks: true (test 1). And when locks are released while the cache is already at its limit, the memory is freed instead of cached: this is the sentence I could not reproduce (tests 2 and 3).

The probe

;WITH lm AS
(
    SELECT SUM(pages_kb) AS lock_manager_kb
    FROM sys.dm_os_memory_clerks
    WHERE type = N'OBJECTSTORE_LOCK_MANAGER'
),
si AS
(
    SELECT committed_kb, committed_target_kb
    FROM sys.dm_os_sys_info
),
pc AS
(
    SELECT
        MAX(CASE WHEN counter_name = N'Lock Blocks'                THEN cntr_value END) AS lock_blocks_in_use,
        MAX(CASE WHEN counter_name = N'Lock Blocks Allocated'      THEN cntr_value END) AS lock_blocks_allocated,
        MAX(CASE WHEN counter_name = N'Lock Memory (KB)'           THEN cntr_value END) AS lock_memory_kb,
        MAX(CASE WHEN counter_name = N'Database Cache Memory (KB)' THEN cntr_value END) AS database_cache_kb,
        MAX(CASE WHEN counter_name = N'Free Memory (KB)'           THEN cntr_value END) AS free_memory_kb
    FROM sys.dm_os_performance_counters
    WHERE object_name LIKE N'%Memory Manager%'
)
SELECT
    SYSDATETIME()  AS measured_at,
    lm.lock_manager_kb / 1024  AS lock_manager_mb,
    si.committed_kb / 1024  AS committed_mb,
    si.committed_target_kb / 1024 AS target_mb,
    CAST(100.0 * lm.lock_manager_kb / si.committed_target_kb AS DECIMAL(5, 1)) AS lock_manager_pct,
    pc.lock_blocks_in_use,
    pc.lock_blocks_allocated,
    pc.lock_blocks_allocated - pc.lock_blocks_in_use AS lock_blocks_cached,
    pc.database_cache_kb / 1024 AS buffer_pool_mb,
    pc.free_memory_kb / 1024 AS free_mb
FROM lm
CROSS JOIN si
CROSS JOIN pc;

How to read it:

• lock_manager_mb is the whole lock manager memory: locks in use plus cached structures. On an idle instance, it is the size of the cache. This is the measure that matters.
• committed_mb is what SQLOS has taken from the operating system so far. target_mb is what it is allowed to take. On an instance that has been running for a while, the two are equal. On a freshly restarted lab, they are far apart and nothing in the percentages makes sense until they meet.
• lock_manager_pct is the clerk against the target. The 20 % and the 60 % are read on this column.
• buffer_pool_mb and free_mb show who pays for the cache, and who gets the memory back.
• lock_blocks_in_use, lock_blocks_allocated and their difference lock_blocks_cached are the performance counters on lock blocks. They are coherent on a freshly started instance (3050 allocated, 0 in use) and stop refreshing after a lock storm on every 2025 build I tested: 6 624 047 allocated blocks reported while the clerk held 1 MB. They are here so that you can see it. Do not use them as evidence, sys.dm_tran_locks is the truth for locks held, the clerk for memory.

The lab

SQL Server 2025 Developer Edition, CU5 (17.0.4045.5) first, then CU9 (17.0.5005.3), on virtual machines of a Proxmox lab and on a Nutanix environment. One SQL Server 2022 instance for comparison, in test 5. The SQLOS target is 4096 MB unless stated otherwise (2048 MB in test 4), with min server memory and max server memory pinned to the same value.

A database with one narrow table. Lock escalation is disabled on the table: we want millions of row locks, not one table lock. This is the scenario the documentation describes at table level instead of trace flag 1211:

CREATE DATABASE LockCacheDemo
GO
ALTER DATABASE LockCacheDemo SET RECOVERY SIMPLE
GO
USE LockCacheDemo
GO
CREATE TABLE dbo.T_Locks
(
    id     INT      NOT NULL,
    filler CHAR(20) NOT NULL CONSTRAINT DF_T_Locks_filler DEFAULT ('x'),
    CONSTRAINT PK_T_Locks PRIMARY KEY CLUSTERED (id)
)
GO
INSERT INTO dbo.T_Locks WITH (TABLOCK) (id)
SELECT value
FROM GENERATE_SERIES(1, 30000000)
GO
ALTER TABLE dbo.T_Locks SET (LOCK_ESCALATION = DISABLE)
GO

Thirty million rows, about 1 GB. Nothing in this block touches the lock manager: the TABLOCK on the insert takes one table lock to load fast. This block only provides lockable rows and removes the mechanism that would stop us from locking them one by one.

The workload (session A):

USE LockCacheDemo
GO
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
GO
DECLARE @rows INT = 1000000;

BEGIN TRANSACTION

SELECT COUNT(*) AS rows_locked
FROM (
    SELECT TOP (@rows) id
    FROM dbo.T_Locks WITH (ROWLOCK)
    ORDER BY id
) AS x

-- Leave the transaction open. Measure from session B. Then:
-- ROLLBACK TRANSACTION

Three different settings:

From a second session (session B): did we get row locks and did escalation stay out?

SELECT resource_type, request_mode, COUNT(*) AS lock_count
FROM sys.dm_tran_locks
WHERE request_session_id = 65   -- The session holding the locks
GROUP BY resource_type, request_mode
ORDER BY lock_count DESC;

One shared lock per row (KEY, the row of a clustered index). One intent lock per page, about 245 rows per page. One intent lock on the table and no OBJECT S: no escalation.

Second check, the cost of a lock. Measured on a restarted instance each time so that nothing was cached before the run:

DECLARE @spid    INT    = 65;  -- session A
DECLARE @idle_kb BIGINT = 0;   -- clerk before the transaction

SELECT
    l.lock_count,
    l.key_locks,
    c.lock_manager_kb / 1024                                  AS lock_manager_mb,
    (c.lock_manager_kb - @idle_kb) * 1024.0 / l.lock_count    AS bytes_per_structure,
    (c.lock_manager_kb - @idle_kb) * 1024.0 / l.key_locks     AS bytes_per_row_lock
FROM (
    SELECT COUNT_BIG(*)                                            AS lock_count,
           SUM(CASE WHEN resource_type = N'KEY' THEN 1 ELSE 0 END) AS key_locks
    FROM sys.dm_tran_locks
    WHERE request_session_id = @spid
) AS l
CROSS JOIN (
    SELECT SUM(pages_kb) AS lock_manager_kb
    FROM sys.dm_os_memory_clerks
    WHERE type = N'OBJECTSTORE_LOCK_MANAGER'
) AS c;

For example :

Four runs, the same figure: about 195 bytes per row lock, everything included. Twice the 96 bytes of the documentation (64 for the lock block, 32 per owner block).

Test 1: The 60 % wall has not moved

The name of the option invites a misunderstanding: it is not a limit on locks. Option at its default of 20 and @rows = 70 000 000 far beyond what 60 % of the target can hold (Target at 4968 MB for this run). The lock manager climbs.

The probe shows 20 %, then 24 % without any escalation (the table option prevents it), then 40 %, 50 % and the statement dies:

Severity 19: the statement is terminated and the transaction is rolled back. The same message lands in the SQL Server error log and in the Windows application log.

Option at its default of 20 and @rows = 70 000 000, more than the 30 million rows of the table and far beyond what 60 % of the target can hold (Target at 4968 MB for this run): the wall is at about 16 million row locks.

This test also settled which memory figure the percentages refer to. On the last probe before the error, the clerk was at 2979 MB: 60 % of the 4968 MB target, and 70 % of the 4233 MB SQLOS had committed at that moment. A 60 % limit on the committed memory would have fired about 440 MB earlier, at 2540 MB. The 60 % limit, and with it the 20 % one, are percentages of the target, not of the memory committed so far. This is why the probe divides by committed_target_kb.

Test 2: Releasing the locks frees nothing

This is the test I wrote this post for. Option at 20. Target at 4096 MB for this run, so the cache ceiling is 819 MB. Nine million locks:

buffer_pool_mb is the Database Cache Memory (KB) counter: the data and index pages in memory. The 349 MB are the pages of the nine million rows the SELECT had to read to lock them, nothing else touched the buffer pool. sys.dm_tran_locks is empty, the session holds nothing and the clerk has not moved. 1677 MB of lock structures are sitting in the cache, twice the configured ceiling. The ROLLBACK did not free anything and neither did the minutes after it.

I ran this on CU5 and on CU9, with 5, 6, 9 and 12 million locks with targets of 2 048 and 4 096 MB. Same result every time. The last run, 12 million locks on CU9 left 2193 MB in the cache, 54 % of the target and they were still there hours later.

Test 3: Later transactions free nothing either

There is a second way to read the sentence of the documentation. Not the release of the storm but the releases that follow: once the cache is above its limit, every structure released by a later transaction should be freed instead of cached and the cache would erode with the lock activity of the workload, down to 20 %. “Large concurrent query workloads” as the documentation puts it.

A first attempt with UPDATE TOP (1000000) … SET filler = ‘y’ and a COMMIT, repeated, lowered the clerk by a few MB per pass. Not conclusive: without a ROWLOCK hint, the engine is free to take page locks for an update of this size and then only a few thousand structures travel through the cache at each pass.

So the proper version, with row locks guaranteed and verified. Cache at 40.9 % of the target after a storm, option at 20, and short transactions in a loop:

USE LockCacheDemo
GO

SET NOCOUNT ON;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;

DECLARE @i INT = 0, @n INT;

WHILE @i < 100
BEGIN
    BEGIN TRANSACTION;
    SELECT @n = COUNT(*)
    FROM (SELECT TOP (200000) id
          FROM dbo.T_Locks WITH (ROWLOCK)
          WHERE id > (@i % 50) * 200000
          ORDER BY id) AS x;
    COMMIT;            -- 200 000 structures released
    SET @i += 1;
END

One hundred transactions, 200,000 row locks each, verified in sys.dm_tran_locks while the loop ran: twenty million lock releases, more than four times the 4.6 million structures sitting above the ceiling (858 MB).

The clerk did not move. Twenty million releases while the cache was more than twice its limit and the memory stayed. Whatever “when locks are released, memory is freed rather than cached” describes, it is not something I can observe with row locks, on CU5 or CU9, with the option at its default.

Test 4: What does shrink the cache

Two things did, during the week. None of them is the sentence of the documentation. Memory demand, once. Target at 2 048 MB, option at 20, six million locks, ROLLBACK: 1 117 MB in the cache, 54.6 %. Then a NOLOCK scan of a 17 GB table which takes one schema stability lock and nothing else but needs every page in the buffer pool:

The scan pushed committed_mb up to the target at that moment the buffer pool still wanted pages, SQLOS had nothing left to take from the operating system and it took from the lock manager: 785 MB given back, the clerk under the ceiling. Not one lock released in between. That looked like the mechanism. It was not reproducible either: on CU9 with the cache at 2 193 MB and the instance already at its 4 096 MB target, the same 17 GB scan left the cache at 2 190 MB. sys.dm_os_ring_buffers tells why: the memory broker that governs the lock manager cache (MEMORYBROKER_FOR_CACHE, which is where the object store lives, not MEMORYBROKER_FOR_STEAL) kept sending it a target of 2 809 MB, 68 % of the target with SHRINK notifications of a few MB. There is no 20 % anywhere in the broker’s arithmetic.

DBCC FREESYSTEMCACHE, instantly. The documentation only shows ‘ALL’, which releases the unused entries of every cache, plan cache included, at the price of recompiling every plan afterwards. It also says that most caches can be freed individually. The lock manager cache is one of them:

SELECT name, pages_kb / 1024 AS mb
FROM sys.dm_os_memory_clerks
WHERE name LIKE N'Lock Manager%'; 

DBCC FREESYSTEMCACHE('Lock Manager : Node 0');

Before :

After :

Probe right after, on the 1677 MB cache left by test 3: 8 MB in the clerk, and the memory back in free_mb. The cache is releasable at any time in one instruction. What is missing on these builds is not the ability to release it, it is the trigger.

Test 5: SQL Server 2022 does the same without the option

If the option changed something, the comparison with a version that does not have it should show it. Same table, same workload, @rows = 70000000 for the wall and a value under the wall for the clean ROLLBACK, on SQL Server 2022 and on SQL Server 2025 CU5, CU9 with the option at 20:

Two things in this table. The background release exists on 2022, without any option, and it brings the cache down to 17.6 %, which is where the 20 % ceiling of 2025 would put it. And 2025 is not faster than 2022 at giving the memory back.

What the tests say

Put together, on SQL Server 2025 CU5 and CU9 with one big transaction and then normal lock activity:

• Lock memory, cache included, can grow up to 60 % of the target. Beyond that, error 1204. Confirmed and unchanged by the option.
• When the storm is released, nothing is freed. The structures go to the cache above the configured ceiling and stay there for hours.
• The lock activity that follows does not free them either: twenty million row-lock releases above the ceiling, no measurable change.
• The cache does come down, slowly, through the generic SQLOS mechanisms (background release, memory broker rebalancing when the instance is at its target), the same ones SQL Server 2022 has. On 2025, they are not faster, and the 20 % figure does not appear in the broker’s targets.
• DBCC FREESYSTEMCACHE(‘Lock Manager : Node N’) empties the cache immediately on any version I tried.

So the documented behavior, “when locks are released, memory is freed rather than cached if the lock manager cache already reached its configured limit”, is not something I could reproduce. Not on CU5, not on CU9, not with the release of the big transaction, not with twenty million releases afterwards. The option is there, it accepts 20 to 60, value_in_use says 20 and the cache sits around 40% for as long as I care to watch.

Two caveats, because this is a lab and not a support case. One access pattern: a single transaction holding millions of row locks then released. Virtual machines with 2 and 4 GB targets. Developer Edition. It is possible that the mechanism needs a condition I did not create and that “large concurrent query workloads” means something more specific than what I ran. If you reproduce the protocol above and see the cache come back under 20 % after a release, I want to hear about it. If you do not, you have the same question I have.

A summary

• Released locks are not freed, they are cached by the lock manager. A row lock costs about 195 bytes all included.
• The thresholds (24 % escalation, 60 % hard limit, 20 % cache ceiling) are read against committed_target_kb, not committed_kb. On a lab, set min and max server memory before measuring anything.
• CU5 adds max lock manager cache memory (%): 20 by default, 20 to 60 allowed, a restart to take effect (is_dynamic = 0).
• The 60 % wall and error 1204 are unchanged by the option.
• On CU5 and CU9, releasing the locks does not bring the cache back under the ceiling and neither do twenty million later releases. The documented behavior was not reproducible on my lab.
• The cache does shrink through the generic SQLOS mechanisms slowly as on SQL Server 2022 and the memory broker targets it well above 20 %.
• DBCC FREESYSTEMCACHE(‘Lock Manager : Node N’) empties it instantly (on any version I tried). It is the remedy that works.
• The lock block performance counters stop refreshing on these builds. Use the clerk and sys.dm_tran_locks.

Thank you. Amine Haloui

L’article SQL Server 2025 CU5: max lock manager cache memory (%): what I could reproduce and what I could not est apparu en premier sur dbi Blog.

That was more or less the question Roger Schönmann and I were asking ourselves while staring at two execution plans on a SQL Server 2025 lab instance. The first query scanned the whole Users table of the StackOverflow2010 database on a single thread and took five seconds (elapsed time). The second one scanned exactly the same pages, with a WHERE clause on Location, got a parallel plan and finished in under three hundred milliseconds.

The obvious reading is that parallelism made the difference, and that the optimizer made a poor choice by keeping the first query serial. The numbers ended up saying the opposite, including once I had forced the parallel plan by hand.

The two queries

Everything below ran on SQL Server 2025, cost threshold for parallelism left at its default of 5 (for this lab only ). The Users table holds 299’398 rows on 7’401 pages. Each query ran on a cold buffer pool, so both paid the same disk reads.

CHECKPOINT;

DBCC FREEPROCCACHE; -- on a lab only
DBCC DROPCLEANBUFFERS;  -- on a lab only

SET STATISTICS IO, TIME ON;

SELECT * FROM dbo.Users;

SELECT * FROM dbo.Users
WHERE Location LIKE '%Denmark%';

The read-ahead counts match to the page. The leading wildcard makes the predicate non-sargable, so the second query cannot seek and has to read every row to test it, exactly like the first one. The filter only reduces the rows returned.

The serial query needs about one second of CPU in total, so even a perfect parallel plan could only shave a fraction off it.

How the optimizer decides

The cost threshold for parallelism is only an entry point when the query planner is building the tree of possible execution plans. When the serial plan costs more than the threshold, the optimizer also builds a parallel alternative, then keeps whichever of the two is cheaper.

The costing of the parallel alternative is explained in detail by Adam Machanic: CPU costs are divided by the degree of parallelism, I/O costs are not divided at all, and the exchange operators added to the plan come with a cost of their own.

The plan XML of the serial query gives everything needed to do the arithmetic.

<OptimizerHardwareDependentProperties ... EstimatedAvailableDegreeOfParallelism="2" ... />
<RelOp AvgRowSize="4468" EstimateCPU="0.329495" EstimateIO="5.46609"
       EstimateRows="299398" PhysicalOp="Clustered Index Scan"
       EstimatedTotalSubtreeCost="5.79558" ... >

Around 90% of the scan cost is I/O, which the parallel plan keeps as is. The part that can shrink is 0.33, divided by the two threads the optimizer assumes for costing on this instance, so the best a parallel scan can hope to save is roughly 0.16. In exchange, the plan needs a Gather Streams operator to bring every row back to a single thread, and here that means 299’398 rows the optimizer believes to be 4’468 bytes wide on average, mostly because AboutMe is an nvarchar(max).

There was no NonParallelPlanReason in the XML, the optimization level was FULL, and nothing in the plan pointed to an inhibitor. The optimizer looked at the parallel alternative and turned it down on cost.

Forcing the parallel plan anyway

Understanding the costing did not stop me from wanting to see the parallel plan run. Three attempts failed.

OPTION(QUERYTRACEON 8649, RECOMPILE) compiled with trace flag 8649 listed under IsCompileTime="true", and the plan stayed serial with no NonParallelPlanReason. I cannot say with certainty why the flag did not win here. USE HINT('ENABLE_PARALLEL_PLAN_PREFERENCE') gave the same result.

Adam Machanic‘s make_parallel() function, cross applied to the query as his article describes, produced a spectacular plan with a Gather Streams estimated at 549’756’000’000 rows. The optimizer parallelised the fake branch of the function, closed the parallel zone before its aggregate, and left the scan of Users on the outer serial Nested Loops.

SELECT x.*
FROM dbo.make_parallel() AS mp
CROSS APPLY
(
    SELECT * FROM dbo.Users
) AS x;

What finally worked was going after the estimate instead of the plan. If the Gather Streams is what the optimizer refuses to pay for, it only has to believe that almost no rows will cross it. So I gave it a filter it judges extremely selective, and that removes nothing at runtime:

declare @impossiblevalue int = 0;

SELECT * FROM dbo.Users WHERE reputation >= @impossiblevalue
OPTION (OPTIMIZE FOR (@impossiblevalue = 1000000));

Finally, some parallelism has been introduced!

The statistics show nothing anywhere near a million reputation points, so the plan is compiled for an estimate of 45 rows (based on the density vector). At runtime @impossiblevalue is 0 and every one of the 299’398 users comes back. The optimizer, fooled into seeing a nearly free Gather Streams, finally picks the parallel plan.

And let’s see the main wait statistics of both executions :

CXPACKET is the price of the streams we created by introducing parallelism. 4 producer threads fill packets for a single consumer, which can only hand rows to SSMS as fast as SSMS accepts them. When the exchange buffers are full, the producers wait. The 21 seconds are summed across threads and the wide rows mean few rows per packet, hence the 60’000 waits.

PAGEIOLATCH_SH also grew, from 41 ms to 2’237 ms. Several threads competing for pages of the same table is a plausible explanation, but I have no real proof of it.

What to take from this

For a query that reads the whole table and returns the whole table, most of the estimated cost is I/O, and the optimizer never divides I/O between threads. On Users, that left ~90 percent of the scan cost untouched by parallelism, and only the remaining CPU share, about 0.33, available to be split. Against that small saving, the parallel plan has to pay for funnelling every row through a single exchange, and on this table the exchange cost more than the plan could save. Row count does not change the outcome, since the I/O, the CPU and the exchange all grow with it.

What makes the optimizer choose a parallel plan is the estimated number of rows crossing the exchange: the forced plan got there with an integer comparison that costs nothing, simply because the estimate dropped to 45 rows. What makes a parallel plan actually faster is something else, real CPU work per row to spread across threads, and few rows coming out at the end. The LIKE query has both, which is why it wins. The forced plan had neither, and it burned almost three times the CPU.

L’article SQL Server: why can’t I read the full table in parallel??? est apparu en premier sur dbi Blog.

Building the test

One table, five million rows, and two indexes holding the same two columns in opposite order. The distribution is deliberately skewed, the way real order tables always are.

CREATE TABLE dbo.Orders (
    OrderID     INT IDENTITY(1,1) NOT NULL PRIMARY KEY CLUSTERED,
    Status      VARCHAR(12)   NOT NULL,
    CustomerID  INT           NOT NULL,
    OrderDate   DATETIME2(0)  NOT NULL,
    Amount      DECIMAL(10,2) NOT NULL
);
GO

INSERT INTO dbo.Orders (Status, CustomerID, OrderDate, Amount)
SELECT TOP 3000000
    'Shipped',
    ABS(CHECKSUM(NEWID())) % 100000 + 1,
    DATEADD(MINUTE, -ABS(CHECKSUM(NEWID())) % 1000000, SYSDATETIME()),
    ABS(CHECKSUM(NEWID())) % 50000 / 100.0
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

INSERT INTO dbo.Orders (Status, CustomerID, OrderDate, Amount)
SELECT TOP 1500000
    'Pending',
    ABS(CHECKSUM(NEWID())) % 100000 + 1,
    DATEADD(MINUTE, -ABS(CHECKSUM(NEWID())) % 1000000, SYSDATETIME()),
    ABS(CHECKSUM(NEWID())) % 50000 / 100.0
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

INSERT INTO dbo.Orders (Status, CustomerID, OrderDate, Amount)
SELECT TOP 500000
    'Cancelled',
    ABS(CHECKSUM(NEWID())) % 100000 + 1,
    DATEADD(MINUTE, -ABS(CHECKSUM(NEWID())) % 1000000, SYSDATETIME()),
    ABS(CHECKSUM(NEWID())) % 50000 / 100.0
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

CREATE NONCLUSTERED INDEX IX_Status_Customer ON dbo.Orders (Status, CustomerID);
CREATE NONCLUSTERED INDEX IX_Customer_Status ON dbo.Orders (CustomerID, Status);
GO

UPDATE STATISTICS dbo.Orders WITH FULLSCAN;
GO

Status holds three distinct values across five million rows and CustomerID holds a hundred thousand. Anyone applying the textbook rule puts CustomerID first, since it slices the table thirty-three thousand times more finely, and writes off IX_Status_Customer as a beginner’s mistake.

Both indexes cover a query that selects only OrderID, because the clustering key sits in every nonclustered leaf. That keeps key lookups out of the read counts and leaves nothing but index access to measure.

Run the test: both columns in the WHERE clause

Let’s run the same query against both indexes, with equality predicates on Status and CustomerID, and compare what comes back.

SET STATISTICS IO ON;

SELECT OrderID
FROM dbo.Orders WITH (INDEX(IX_Status_Customer))
WHERE Status = 'Cancelled' AND CustomerID = 42731;

Table 'Orders'. Scan count 1, logical reads 3, physical reads 0

SELECT OrderID
FROM dbo.Orders WITH (INDEX(IX_Customer_Status))
WHERE Status = 'Cancelled' AND CustomerID = 42731;

Table 'Orders'. Scan count 1, logical reads 3, physical reads 0

Three logical reads each. Same plan shape, same cost, same everything, from two indexes whose leading columns differ in cardinality by a factor of thirty-three thousand.

The seek predicates settle it: both say Prefix, and neither plan carries a residual Predicate section, which means both indexes reach the rows with a single descent on the full composite key rather than filtering one column after the other :

The word that matters is Prefix, and it appears on both sides. SQL Server uses that label when the seek keys form a leading portion of the index key and can be resolved in a single descent through the tree.

This is the heart of the whole thing. A composite index does not store two sorted columns side by side, waiting to be consulted in sequence. It stores one sorted key made of both columns, and Cancelled | 42731 occupies exactly one position inside it. The engine descends once and arrives, and the picture most of us carry, in which the engine narrows the search using the first column, hands a smaller set of rows over to the second and narrows it again, describes something SQL Server never does.

That mental picture is precisely what makes the selectivity rule feel so obviously right. Sort by the column that eliminates the most rows first, the reasoning goes, and the second column has less work left to do. It sounds like common sense, and the only trouble with it is that it describes a two-stage filter which the seek predicates above show to be nowhere in either plan, because the filtering it imagines was already finished the moment the descent landed.

Deep-dive into the nonclustered B-trees

The plans agree. Now let’s see whether the physical structures agree too.

SELECT
    index_level,
    page_count,
    record_count,
    avg_page_space_used_in_percent
FROM sys.dm_db_index_physical_stats(
        DB_ID(),
        OBJECT_ID('dbo.Orders'),
        INDEXPROPERTY(OBJECT_ID('dbo.Orders'), 'IX_Status_Customer', 'IndexID'),
        NULL,
        'DETAILED')
ORDER BY index_level DESC;

SELECT
    index_level,
    page_count,
    record_count,
    avg_page_space_used_in_percent
FROM sys.dm_db_index_physical_stats(
        DB_ID(),
        OBJECT_ID('dbo.Orders'),
        INDEXPROPERTY(OBJECT_ID('dbo.Orders'), 'IX_Customer_Status', 'IndexID'),
        NULL,
        'DETAILED')
ORDER BY index_level DESC;

IX_Status_Customer:

IX_Customer_Status:

Start with the number that ties back to test 1. Both indexes have three levels, and both queries reported three logical reads. Root page, intermediate page, leaf page. The execution plan and the physical structure confirm each other through two completely independent measurements, and neither leaves room for a second lookup step hiding somewhere. Five million rows on both sides, same depth, same cost, and the number of distinct values in the leading column changes neither one.

Now read the tables diagonally. The record_count of any level equals the page_count of the level beneath it: 57 records at level 2 against 57 pages at level 1, then 15’605 records at level 1 against 15’605 pages at level 0. The second index does the same with its own figures, 58 and 58, then 15’589 and 15’589. An upper level therefore holds one entry per child page. One per page, whatever that page happens to contain, which is why the size of these levels tracks the number of pages underneath them and stays completely indifferent to how many distinct values live in the column you chose to lead with.

And that brings us to the comparison the whole article was built for:

Inside the root pages

The DMV tells us each root holds fifty-seven and fifty-eight entries. Let’s look at what those entries actually are.

SELECT
    allocated_page_file_id,
    allocated_page_page_id,
    page_type_desc,
    page_level
FROM sys.dm_db_database_page_allocations(
        DB_ID(),
        OBJECT_ID('dbo.Orders'),
        INDEXPROPERTY(OBJECT_ID('dbo.Orders'), 'IX_Status_Customer', 'IndexID'),
        NULL,
        'DETAILED')
WHERE page_type_desc = 'INDEX_PAGE'
ORDER BY page_level DESC;

SELECT
    allocated_page_file_id,
    allocated_page_page_id,
    page_type_desc,
    page_level
FROM sys.dm_db_database_page_allocations(
        DB_ID(),
        OBJECT_ID('dbo.Orders'),
        INDEXPROPERTY(OBJECT_ID('dbo.Orders'), 'IX_Customer_Status', 'IndexID'),
        NULL,
        'DETAILED')
WHERE page_type_desc = 'INDEX_PAGE'
ORDER BY page_level DESC;

--Run once per index, using the page_level 2 page id returned above
DBCC TRACEON(3604);
DBCC PAGE('IndexOrderDemo', 1, <root_page_id_for_both_index>, 3);

IX_Status_Customer:

IX_Customer_Status:

Several things show up in these dumps, and each of them says the same thing about how the key is treated.

Row 0 carries NULL in every key column, which means an index record holding no key whatsoever, pointing at the leftmost child page and covering every row that sorts below the first boundary anywhere in the page.

Every other entry carries all three key columns, OrderID included, because the clustering key forms part of every nonclustered key. No entry ever names a single column. Each one identifies one specific row out of five million, which is exactly what a boundary between two pages has to do.

In IX_Status_Customer, the same status value appears in entry after entry. Half a million Cancelled rows span roughly 1’560 leaf pages and therefore about six intermediate pages, so the value contributes six separate boundaries.

What about statistics?

Navigation is one thing, description is another. Run DBCC SHOW_STATISTICS on both indexes and the density vectors tell the story of this whole article in five numbers:

DBCC SHOW_STATISTICS('dbo.Orders', 'IX_Status_Customer');

DBCC SHOW_STATISTICS('dbo.Orders', 'IX_Customer_Status');

The single-column entries are worlds apart, 0.3333333 for Status against 1E-05 for CustomerID. The two-column entries are identical to the last digit, 3.340538E-06 on both, because the density of the full prefix counts the same combinations whichever way round you write them. Furthermore, both previous execution plans above show 7 of 17, seven rows actually returned against seventeen expected, and seventeen is not a coincidence: 5’000’000 multiplied by the two-column density of 3.340538E-06 gives 16.7. Since that density is identical in both indexes, so is the estimate.

The histograms for the first column have nothing in common. IX_Status_Customer gets three steps with exact row counts, 500’000 Cancelled, 1’500’000 Pending, 3’000’000 Shipped. IX_Customer_Status gets 21 steps across a hundred thousand customer numbers, with around 50 rows per step. A histogram only ever describes the leading column, so each index is blind to the other’s.

That matters the moment a query supplies one column instead of both, and it is worth knowing before you assume the two indexes are interchangeable for every purpose. It changes nothing for equality on the full key, which is what this article measured.

What to take from this

For equality predicates covering the full key, the two indexes are interchangeable. Three logical reads each, three levels each, and root pages holding 57 entries against 58, from two indexes whose leading columns sit thirty-three thousand apart in cardinality. Brent was right, and the plan, the DMV and the pages all say so independently.

The reason sits in the seek predicates. Prefix on both sides, no residual predicate on either, one descent to one composite key. SQL Server treats the columns as a single sorted value, so there is no first column doing the heavy lifting and no second column mopping up.

None of which makes column order free. Drop the leading column and the two part company immediately: WHERE CustomerID = 42731 alone seeks on one index and scans all 15’605 leaf pages on the other. Add an inequality and only the leading range predicate stays seekable, since a range cannot be resolved by a single descent. The seek finds the boundary and reads forward, everything after that column becomes a residual predicate evaluated row by row, and you can watch it appear as a Predicate section under the seek that was absent from both plans above.

So order your key columns by what your queries supply. Equality columns first, then the single range predicate that eliminates the most rows, because only the first of them gets a real seek. That is the one place selectivity earns its keep.

One last thing worth saying plainly: this article compared two indexes on a single query, and no index ever exists to serve a single query. The right order for IX_Status_Customer or IX_Customer_Status depends on everything else your application runs against that table. Which columns appear alone, which appear together, which carry ranges. That is the analysis worth doing, and counting distinct values will never be a substitute for it.

Next time you are ordering the columns of a composite index, write down the two or three query shapes that actually matter in your workload, check which of them the key prefix can serve, and let that decide the order. If you find yourself counting distinct values instead, stop, because you are answering a question nobody asked.

L’article SQL Server: Stop ordering index columns by selectivity est apparu en premier sur dbi Blog.

The answer turns on how SQL Server compares uniqueidentifier values, which is not explicitly documented. So it has to be measured.

Context

To showcase it, run this on any instance. No database required.

SELECT g
FROM (VALUES
    (CAST('ffffffff-ffff-ffff-ffff-000000000000' AS uniqueidentifier)),
    (CAST('00000000-0000-0000-0000-000000000001' AS uniqueidentifier))
) v(g)
ORDER BY g;

The value starting with ffffffff comes back first, ahead of the one that is almost entirely zeros.

That behaviour is intentional, and Microsoft covers it in a sentence that is easy to overlook: for uniqueidentifier, ordering is not implemented by comparing the bit patterns of the two values. The documentation stops there. It tells you the comparison is something other than a plain byte scan, then leaves the actual sequence unstated.

The omission has become expensive. Developers are now bringing UUIDv7, the time-ordered identifier standardised in RFC 9562, on the reasonable assumption that a leading timestamp turns scattered inserts into an append pattern. That holds where a database compares UUIDs lexicographically over the RFC byte sequence. SQL Server uses a different order, and nothing about the identifier warns you.

Since the order is undocumented, the only way to establish it is to measure it.

How the bytes are stored

A uniqueidentifier is a 16-byte value. The five hyphen-separated groups you see on screen are a display convention. Storage follows the Windows GUID structure, which is mixed-endian: the first three groups are byte-reversed on disk, while the last two keep the order you read them in.

SELECT CAST(CAST('01020304-0506-0708-090a-0b0c0d0e0f10' AS uniqueidentifier) AS binary(16));

Every byte here is distinct, so both halves of the rule are visible at once. The first group 01020304 is stored 04 03 02 01, the second 0506 becomes 06 05, the third 0708 becomes 08 07. The fourth and fifth groups appear exactly as written.

The comparison order

The ADO.NET documentation supplies part of the answer. SqlGuid implements CompareTo to match SQL Server behaviour, treating the last six bytes as the most significant, whereas System.Guid evaluates all sixteen.

Testing the rest against a live instance produces a sequence that held for every value I tried:

SQL Server reorders the sixteen stored bytes as 10-15, 8-9, 6-7, 4-5, 0-3, then compares them left to right.

A single permutation followed by an ordinary binary comparison, with no special handling per group.

Applied to the two values above:

A = ffffffff-ffff-0010-0000-aaaaaaaaaaaa
B = ffffffff-ffff-2000-0000-aaaaaaaaaaaa

stored     A : FFFFFFFF FFFF 1000 0000 AAAAAAAAAAAA
           B : FFFFFFFF FFFF 0020 0000 AAAAAAAAAAAA

reordered  A : AAAAAAAAAAAA 0000 1000 FFFF FFFFFFFF
           B : AAAAAAAAAAAA 0000 0020 FFFF FFFFFFFF

The first eight bytes are identical in both keys, the ninth settles the comparison, and the remaining seven are never examined.

The six bytes read first are stored positions 10 to 15. In an RFC 4122 UUIDv1 those hold the node field, historically derived from the MAC address, though implementations are free to substitute a random value there.

Testing it on 10,000 values

Hand-picked examples illustrate a rule without showing that it generalises. To check that, rebuild the proposed key in T-SQL and compare the order it predicts against the engine’s own.

WITH G AS (
    SELECT TOP (10000) g = NEWID() FROM sys.all_columns a CROSS JOIN sys.all_columns b
), B AS (
    SELECT g, b = CAST(g AS binary(16)) FROM G
), P AS (
    SELECT g, Cle = SUBSTRING(b,11,6) + SUBSTRING(b,9,2) + SUBSTRING(b,7,2)
                  + SUBSTRING(b,5,2)  + SUBSTRING(b,1,4)
    FROM B
), R AS (
    SELECT R1 = ROW_NUMBER() OVER (ORDER BY g),
           R2 = ROW_NUMBER() OVER (ORDER BY Cle)
    FROM P
)
SELECT Mismatches = SUM(CASE WHEN R1 <> R2 THEN 1 ELSE 0 END) FROM R;

Zero mismatches on 10,000 values. Substituting the naive key, sixteen bytes read straight through from left to right, returns 10,000 mismatches on the same set, which at least confirms the test discriminates between the two models.

That is strong evidence, but it stops short of proof. Repeated runs, with fresh values each time, returned zero mismatches every time. The implementation itself is undocumented, and what I ran covers one version of SQL Server (2022).

Where UUIDv7 puts its timestamp

RFC 9562 builds UUIDv7 around a 48-bit big-endian Unix timestamp in milliseconds, placed in the leading bytes. The remaining fields carry version and variant bits along with random data, optionally including a sub-millisecond fraction or a counter to improve monotonicity inside a single millisecond.

TTTTTTTT - TTTT - 7RRR - VRRR - RRRRRRRRRRRR
|_____________|                              48-bit timestamp

That timestamp occupies the first two textual groups, so in storage its bytes land in positions 0 to 3 and 4 to 5, both byte-reversed. Positions 4 to 5 are read only after 10 to 15, 8 to 9 and 6 to 7, and positions 0 to 3 come last of all.

Which means the fields SQL Server examines first contain version bits, variant bits and random data. Independently generated values separate on those long before the timestamp is reached, so for a clustered index taking single-row inserts the append behaviour UUIDv7 is known for simply does not materialise.

Measured on 25,000 inserts

Three tables, identical apart from the clustered key, FILLFACTOR 100, 25,000 rows inserted one row per statement. Bulk loading can hide the effect entirely. When SQL Server sets DMLRequestSort on a clustered index insert, it feeds the rows in key order, which promotes sequential writes and avoids page splitting. Whether it does so depends on the estimated row count, the locking hint and a cost decision, so the safest way to reproduce an OLTP pattern is to insert one row per statement.

About one percentage point separates NEWID() from UUIDv7, and the sign changes between runs, so under this workload UUIDv7 buys you nothing at all.

The sequential key holds the same rows in 391 pages rather than 560, roughly 30 % fewer. Since pages are the unit SQL Server reads and caches, a sparse index needs more buffer pool to hold the same rows and more reads to scan them, which puts the cost on memory and I/O well before it shows up as disk space.

A few things are worth knowing if you reproduce this. The measurements come from the sys.dm_db_index_physical_stats function, called in DETAILED mode: avg_page_space_used_in_percent comes back NULL under LIMITED, and passing an index_id while object_id is NULL raises an error rather than returning every index. Read page fullness in preference to fragmentation, since a mid-page split leaves both halves partly empty and fullness is the figure that registers it.

Practical options

The costly combination here is quite specific: a GUID that is random under SQL Server’s comparison order, used as the clustering key, fed by sustained single-row inserts. Change any one of those three and the physical consequences shift. GUIDs still earn their place whenever identity has to be generated away from the database, by clients, by distributed services, or by merge replication, which relies on uniqueidentifier to keep rows distinct across copies.

A sequential clustered key sends every insert to the same last page, and under concurrency those threads all queue on the same PAGELATCH_EX. Microsoft describes what happens next: the insert that triggers a new page holds the latch longer than usual, the queue builds up behind it, and throughput falls off a cliff. OPTIMIZE_FOR_SEQUENTIAL_KEY caps how many threads may queue for the latch, which keeps throughput steadier without removing the contention.

The useful question, then, is where the GUID sits in the physical design and which of those two costs your workload actually pays.

You can move it off the clustering key while leaving it as the primary key. SQL Server lets you declare PRIMARY KEY NONCLUSTERED on the GUID and cluster on a narrow IDENTITY column instead. The physical result matches a clustered IDENTITY with a unique constraint on the GUID, but the declaration matches the logical model: the GUID identifies the row, the IDENTITY only orders it on disk.

CREATE TABLE dbo.T (
    RowId bigint           IDENTITY NOT NULL,
    Id    uniqueidentifier NOT NULL DEFAULT NEWID(),
    CONSTRAINT PK_T PRIMARY KEY NONCLUSTERED (Id),
    INDEX CX_T CLUSTERED (RowId)
);

Either way the table stays dense and the 16-byte cost is confined to one index. The question is also narrower than it looks, since it applies to OLTP tables taking single-row inserts. On the analytical side a clustered columnstore index changes the storage model entirely, page density in the rowstore sense stops being the right measure, and this is a typical solution for some Data Warehouse use-cases (thanks to Uwe Ricken for the reminder 😉).

You can make it sequential instead. NEWSEQUENTIALID() works because it puts the increasing part where SQL Server reads first. Its documented limits are worth reading before you commit: the values are guessable and unsuitable where privacy matters, it only functions as a column DEFAULT, and the sequence can restart from a lower range after a Windows restart.

Or accept the cost. That is easy on a small table with rare inserts, where a single rebuild holds and the wasted space is trivial anyway, and harder on a busy random-key table that drifts back to the same density (or higher) after every rebuild.

Another solution could be a heap, because it removes the ordering question from the table itself: rows land wherever free space allows, and a nonclustered index on the GUID locates them by RID rather than by the clustering key. It suits tables with little DML. An update that grows a row past its page leaves a forwarding record behind, and the RID then costs an extra read on every lookup that follows it. Deletes have a different drawback: the rows are marked as ghosted, but the emptied pages stay allocated to the table unless the delete takes a table lock or the heap is rebuilt.

Whichever way you go, audit what is already in production. A large uniqueidentifier-keyed index sitting near 69 % fullness deserves a look, since it may be taking values whose insertion order is effectively random for SQL Server, whether they come from NEWID(), UUIDv7 or something else. Density only points somewhere. Check fill factor, insert pattern and rebuild history before drawing a conclusion.

SELECT
    [Schema]      = OBJECT_SCHEMA_NAME(ps.object_id),
    [Table]       = OBJECT_NAME(ps.object_id),
    [Index]       = i.name,
    [FillFactor]  = NULLIF(i.fill_factor, 0),
    Pages         = ps.page_count,
    Fullness      = CAST(ps.avg_page_space_used_in_percent AS decimal(5,1)),
    Fragmentation = CAST(ps.avg_fragmentation_in_percent  AS decimal(5,1))
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, 'SAMPLED') ps
JOIN sys.indexes i ON i.object_id = ps.object_id AND i.index_id = ps.index_id
JOIN sys.index_columns ic ON ic.object_id = i.object_id
                         AND ic.index_id  = i.index_id
                         AND ic.key_ordinal = 1
JOIN sys.columns c ON c.object_id = ic.object_id AND c.column_id = ic.column_id
WHERE ps.index_level = 0
  AND ps.alloc_unit_type_desc = 'IN_ROW_DATA'
  AND ps.page_count > 500
  AND i.type IN (1, 2)
  AND TYPE_NAME(c.system_type_id) = 'uniqueidentifier'
ORDER BY ps.avg_page_space_used_in_percent;

Where that leaves us

An RFC specifies what the bits mean, and nothing in it governs how a storage engine chooses to order them.

UUIDv7 delivers what it promises, a sortable timestamp at the start of the RFC byte sequence, and SQL Server applies the comparison order it has always applied, in which that timestamp carries almost no weight. Both behaviours are internally consistent. Put together, under the workload measured here, they produce no clustered-index locality whatsoever, and the version number in the identifier gives you no hint that the temporal ordering has been lost along the way.

So when the next identifier scheme arrives described as sequential, the version number and the printed form will settle very little. What counts is where the increasing portion ends up once the engine has applied its storage layout and its comparison order.

L’article Why UUIDv7 does not reduce fragmentation in SQL Server est apparu en premier sur dbi Blog.

The problem

Migrating ~5,600 Oracle tables to SQL Server with SSMA. Most load fine; ~100 tables fail Migrate Data with the same error:

ERROR [42S22] [Oracle][ODBC][Ora]ORA-00904:
"SYS_C00004_21081414:28:22$": invalid identifier

The named column exists in no DDL anyone wrote. The [Ora] prefix says Oracle itself is rejecting the query: SSMA built an extraction SELECT naming a column Oracle refuses to resolve. Tellingly, SELECT * and COUNT(*) run fine against the same table, whatever this column is, Oracle is happy to ignore it, but not to be asked for it by name.

Where the column actually comes from

The name is the giveaway: SYS_C00004_21081414:28:22$ is what Oracle calls a column after ALTER TABLE … SET UNUSED COLUMN.

Oracle offers two ways to get rid of a column: a logical delete and a physical one. The physical delete (ALTER TABLE … DROP COLUMN) is the honest one, but on a large table it is very time- and resource-consuming. That’s why people reach for the logical delete instead:

ALTER TABLE table_name SET UNUSED (column_name);

That statement is metadata-only and instant. The column immediately stops being visible to users, and the physical removal is deferred to whenever there is time for it (see Oracle Documentation):

ALTER TABLE table_name DROP UNUSED COLUMNS;
-- on large tables, cap undo growth by checkpointing every N rows:
ALTER TABLE table_name DROP UNUSED COLUMNS CHECKPOINT 250;

To free the original name for reuse, Oracle renames the column to SYS_C<internal column number>_<YYMMDDHH24:MI:SS>$, sets USER_GENERATED to NO, HIDDEN_COLUMN to YES and releases its COLUMN_ID. So the timestamp is not when the column was added, it is the second someone ran SET UNUSED. Ours says 14 August 2021, 14:28:22.

That also explains the error pattern. The operation is one-way and the column is unreadable by design, so naming it gets you ORA-00904 «invalid identifier». SELECT * and COUNT(*) keep working because the column no longer has a COLUMN_ID and is simply excluded from the star. SSMA, however, lists it and builds an explicit column list Oracle then refuses.

Not to be confused with SYS_NC…$. Those are a different animal: virtual columns backing a function-based index or extended statistics.

Bottom line: returning NULL costs you nothing. This is a column its owner already decided to delete, holding data Oracle itself will no longer hand out. There is no information left to lose.

What doesn’t work for the migration

• SSMA setting Ignore hidden system columns = Yes was not making any effect on this use case
• Dropping the column on SQL Server resolves nothing because the error is on the source SELECT, unaffected.
• Dropping it on Oracle could not be done in our scenario because the source is frozen; DDL not allowed.
• Custom select, column removed or bare NULL: SSMA still expects the name in its mapping and fails with “key not present” or “does not match up” before the query ever reaches Oracle.

Find them all first

Before editing anything, get the full list. Discovering the affected tables one failed migration at a time is a waste of an afternoon because the data dictionary already knows.

The reason the columns are findable at all is an asymmetry between two views: an unused column is gone from ALL_TAB_COLUMNS, but still listed in ALL_TAB_COLS with HIDDEN_COLUMN = 'YES'. That second view is what you query:

SELECT owner,
table_name,
column_name,
data_type,
internal_column_id,
TO_DATE(REGEXP_SUBSTR(column_name, '\d{8}:\d{2}:\d{2}'),
'YYMMDDHH24:MI:SS') AS set_unused_at
FROM dba_tab_cols
WHERE hidden_column = 'YES'
AND user_generated = 'NO'
AND REGEXP_LIKE(column_name, '^SYS_C\d+_\d{8}:\d{2}:\d{2}\$$')
-- AND owner = '<SCHEMA_NAME>'
ORDER BY owner, table_name, internal_column_id;

The regex is deliberately strict: it matches only the SET UNUSED naming pattern, so virtual columns and other system-generated names stay out of the result.

One more view is worth a look, as a cross-check:

SELECT owner, table_name, count AS unused_columns
FROM   dba_unused_col_tabs
--WHERE owner = '<SCHEMA_NAME>'
ORDER  BY count DESC, table_name;

DBA_UNUSED_COL_TABS gives the number of unused columns per table. Sorting by that number puts the dangerous tables first: those with two or three hidden columns are the ones where you’ll forget a line in the custom select and be back at square one.

The fix

Keep the hidden column’s name as an alias, but return a literal NULL instead of reading it. SSMA’s mapping finds the name (no “key not present”); Oracle never resolves the real column (no ORA-00904).

• Tools → Project Settings → General → Migration → enable Extended data migration options.

• Data Migration Settings tab → tick Use custom select → replace each hidden-column line with:

 SELECT ...   
    TO_CHAR("<COLUMN_NAME>", 'TM', 'NLS_NUMERIC_CHARACTERS = ''.,''') as "<COLUMN_NAME>",
    NULL as "SYS_C00004_21081414:28:22$"
from <OWNER>.<TABLE_NAME> t

• Migrate Data → 100%. Drop the NULL-filled column(s) on SQL Server in post-migration cleanup.

Takeaway

SYS_C…$ is not an exotic Oracle feature, it’s an ordinary column someone deleted years ago, logically. Oracle keeps the name on file; SSMA finds it, insists on naming it, and Oracle refuses to hand it over. Aliasing a NULL satisfies both, then you drop the column on the target. No source DDL, no external tooling, everything inside SSMA, behind a project setting that’s hidden by default.

L’article Fixing ORA-00904 on Oracle hidden columns during SSMA data migration est apparu en premier sur dbi Blog.

The starting point: one number from a health check

The I/O statistics of the instance (sys.dm_io_virtual_file_stats) reported the following for tempdb hosted on a local volume D:

And the global view of the volume:

Two details frame the whole investigation:

• Reads are excellent. Writes are 19 times over the threshold. The read path is healthy, the write path is not.
• io_stall_write_ms measures the time between I/O submission and completion, queue time included. A high average does not tell us whether each write is slow or whether writes are waiting behind each other.

When SQL Server performs a write operation, the request goes through the following (simplified) path:

What it is not

The 8 data files show nearly identical latencies and write counts (about 15.8 M each). The tempdb round-robin allocation works perfectly. This is not a hotspot, not a single bad file. The whole volume is affected.

The volume D is local to each cluster node. It is not a shared disk (not on the storage array).

Slow media or queuing?

The DMV cannot separate service time from queue time. So we measured the service time directly at idle with WinSAT:

The media is excellent on all four access profiles. The verdict is simple: the 380 ms are queue time, not service time.

A quick calculation confirms it. The DMV counted about 128 M writes over 104 hours of uptime: about 340 writes per second on average. The volume can absorb 50,000. Average utilization: 0.7%. A volume used at 0.7% that shows 376 ms of average latency means one thing: the load is not smooth. It arrives in bursts. During a burst, thousands of I/Os pile up in the queue, each one waits behind the others and since most of the write volume is concentrated in those bursts they dominate the average.

Where the bursts come from

On this instance, the bursts are produced by sort and hash operations that do not fit in their memory grant and spill to tempdb, mainly during data loads and some heavy analytical queries. The workload side of this story (memory grants, parallelism, NUMA topology) is covered in this blog : https://www.dbi-services.com/blog/wait-stats-and-sub-numa-clustering/

In this post, we follow the storage path only: whatever the workload does, a burst of writes should not cost 380 ms per I/O on a volume this fast.

The invisible layer

When SQL Server writes a page to tempdb, the write goes through this chain:

SQL Server > Windows/NTFS > driver (SmartPqi.sys) > Smart Array controller > physical SSDs

Windows never talks to the SSDs. It talks to the RAID controller (an HPE Smart Array P408i-a) which assembles two SAS SSDs into a RAID 1 mirror and presents the result as volume D.

Reference: https://support.hpe.com/connect/s/softwaredetails?language=en_US&collectionId=MTX-3d51e7d6b8674f16&tab=releaseNotes

Here is the key point: every instrument used so far measures through that controller without seeing it. The DMVs measure above it. WinSAT measures above it. Only one question remains open: how is that card configured? And only one tool answers it: the Smart Storage Administrator CLI (ssacli).

Factory settings

ctrl slot=0 show detail
   
   Cache Board Present: True
   Total Cache Size: 2.0
   Cache Status: Not Configured
   Battery/Capacitor Status: OK
   No-Battery Write Cache: Disabled

ctrl slot=0 ld all show detail
   
   Logical Drive: 2          (volume D)
      Fault Tolerance: 1     (RAID 1)
      Caching: Disabled
      LD Acceleration Method: Smart Path

HPE SSD Smart Path is a direct I/O path: requests skip the RAID firmware stack and go straight to the SSDs. It saves a few dozen microseconds per I/O which benefits reads. But it is a per-volume switch and it is mutually exclusive with the controller cache. Smart Path is enabled by default on every SSD array (factory default). It’s reasonable for a read-oriented volume but it was never revisited for a volume hosting tempdb (one of the most write-intensive profiles there is).

The consequence: every write must be applied to both SSDs of the mirror and confirmed before it is acknowledged. There is no absorber anywhere in the chain. When a burst arrives the queue explodes.

Is enabling the write cache safe?

The old advice “do not enable write caching” targets a different cache: the volatile DRAM inside the disks themselves which loses acknowledged writes on power failure. That one stays disabled (Drive Write Cache Policy: Disable).

The controller cache is a different story. Microsoft’s requirement is stable media: an acknowledged write must survive a power failure. This controller qualifies through the flash-backed write cache mechanism:

• On power loss, the battery does not store any data. It powers the cache module for a few seconds just long enough for the controller to copy the DRAM content to the flash NAND chip on the module itself. Flash is non-volatile: the data survives without any power (indefinitely).
• At reboot the controller restores that data and writes it to the SSDs of the volume before accepting any new I/O.
• If the battery ever fails the controller detects it and automatically falls back to write-through.

There is a second safety belt specific to this volume: tempdb is recreated at every instance startup. Even in the worst theoretical scenario, there is no data anyone would come back for.

The possible fix

Three online reversible commands:

ssacli ctrl slot=0 array B modify ssdsmartpath=disable
ssacli ctrl slot=0 ld 2 modify caching=enable
ssacli ctrl slot=0 modify cacheratio=10/90

Why cacheratio=10/90?

This setting splits the controller cache: 10% for reads, 90% for writes. It is not an exotic choice, it is the HPE factory default for a configured cache documented as the best ratio for most workloads.

Reference: https://support.hpe.com/hpesc/public/docDisplay?docId=a00019059en_us&page=GUID-EE28F5A4-ADF5-4E27-81AA-8377A267FFA7.html

Expected result: LD Acceleration Method: Controller Cache on the logical drive, Cache Status: OK on the controller.

To validate we should not rely on the cumulative DMV averages they will stay polluted by history. We should measure deltas:

• WinSAT after the change: service time should stay excellent (nothing was broken).
• sys.dm_io_virtual_file_stats deltas over a defined window covering the load phases.
• PerfMon during the load window: Avg. Disk sec/Write should stay in single digits at the peak of a burst and the queue should drain between bursts.

One expectation to set correctly: the cache absorbs bursts but it does not add throughput. All the bytes still land on the same two SSDs.

Note: these commands have not been implemented. They are proposals only. The change must be reviewed, validated and scheduled by the customer before any implementation.

Local tempdb volumes on FCI nodes: a good idea

Placing tempdb on a local volume in a Failover Cluster Instance is supported since SQL Server 2012 and it is a good design: tempdb is recreated at startup so there is nothing to fail over. It offloads the shared storage and local SSDs deliver excellent performance for one of the hottest write profiles of the instance (our measurements above prove it).

But this choice turns storage health into a per-node responsibility:

• Check the RAID controller configuration on every node. The passive node most likely carries the same factory default. After a failover the problem would silently come back.
• Monitor Battery/Capacitor Status. A dead battery silently disables the write cache and brings the symptom back.

The architecture is right. It just makes your RAID controller part of your database health check.

Thank you. Amine Haloui

L’article When tempdb write latency points below SQL Server est apparu en premier sur dbi Blog.

When a company still refuses to put its sensitive data in the cloud, the reason usually isn’t cost or performance: it’s data sovereignty. In the cloud, someone else is the administrator of the machine; therefore the provider can, in theory, read the data files, see the data being used in memory, or read the backups. The threat model is no longer the external attacker but the privileged insider hosting the database.

As covered previously on my blog Beyond TDE and TLS: Bridging the Data Security Governance Gap in Lower Environments, various encryption methods can protect you. For example, TLS protects data in transit and TDE protects it at rest, but as soon as the engine runs a query, it handles plaintext in memory. So there are three states to protect: at-rest, in-transit, in-use. The gap this article focuses on is the last one.

That is exactly what Always Encrypted (SQL Server) and Queryable Encryption (MongoDB) target. The common principle: encryption and decryption happen client-side, in the driver; the keys never reach the engine. The data stays encrypted at rest, in transit, and during processing. The DBA, the cloud operator, the hypervisor admin: all of them see only cyphertext.

That leaves one question: if the engine sees only cyphertext, how does it answer a WHERE condition? Both database engines do answer it, but through different technical means.

SQL Server: queryability lives in the cyphertext

To demonstrate all this, let’s start by creating a table with two columns, Salary and Department, encrypted deterministically on one side and randomized on the other:

DROP TABLE IF EXISTS dbo.Employees;
CREATE TABLE dbo.Employees (
  Id         INT IDENTITY(1,1) PRIMARY KEY,
  LastName   NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  FirstName  NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  DeptDet    NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be DETERMINISTIC
  DeptRand   NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be RANDOMIZED
  SalaryDet  INT NOT NULL, -- will be DETERMINISTIC
  SalaryRand INT NOT NULL  -- will be RANDOMIZED
);
GO

INSERT INTO dbo.Employees (LastName, FirstName, DeptDet, DeptRand, SalaryDet, SalaryRand) VALUES
('Martin',  'Alice', 'Sales', 'Sales', 55000, 55000),
('Dubois',  'Bob',   'Sales', 'Sales', 48000, 48000),
('Bernard', 'Chloe', 'Sales', 'Sales', 52000, 52000),
('Petit',   'David', 'IT',    'IT',    72000, 72000),
('Durand',  'Emma',  'IT',    'IT',    68000, 68000);

Five rows: three Sales, two IT. The BIN2 collation is required by Always Encrypted (link to documentation), and the master key lives outside the database (Key Vault, certificate store, or HSM).

Column encryption isn’t done in T-SQL, because the engine doesn’t have the keys. It’s driven from the client (here in PowerShell), declaring for each value a deterministic column and its randomized twin:

Import-Module SqlServer -MinimumVersion 22.0.59
$sqlConnectionString = "Data Source=.\LAB2025;Initial Catalog=AEDEMO;Integrated Security=True;Encrypt=False;Trust Server Certificate=False"
$smoDatabase = Get-SqlDatabase -ConnectionString $sqlConnectionString

$encryptionChanges  = @()
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptDet    -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptRand   -EncryptionType Randomized    -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryDet  -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryRand -EncryptionType Randomized    -EncryptionKey "CEK1"

Set-SqlColumnEncryption -ColumnEncryptionSettings $encryptionChanges -InputObject $smoDatabase

In this example, I’m working on my 2025 SQL Server instance, on the AEDEMO database, using the column encryption key CEK1 I created beforehand (itself protected by a column master key stored outside the database).

We check that the engine sees the right type per column:

SELECT c.name, c.encryption_type_desc
FROM sys.columns c
WHERE c.object_id = OBJECT_ID('dbo.Employees');

The deterministic mechanism works like this: same plaintext, same cyphertext (an injective function). On an Always Encrypted-enabled connection with Parameterization for Always Encrypted and parameters for the predicates (never literals), equality works:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptDet AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptDet = @d
GROUP BY DeptDet;

The driver encrypts @d with the same key, the server finds the encrypted values that are identical to the parameter, and the result is correct.

You pay for it in three ways…

First weakness: deterministic encryption can cause a data leak. You don’t need the keys to see it. Just connect without Always Encrypted and read the table: the encrypted columns come out as raw binary.

SELECT FirstName, DeptDet, DeptRand FROM dbo.Employees;

Look at the DeptDet column: Alice, Bob, and Chloe share exactly the same blob, and David and Emma share another. Two distinct values across five rows. The adversary doesn’t know what 0x012536… means, but reads the structure: two departments, one with three people, the other with two, and who goes with whom. The DeptRand column, on the other hand, shows five all-different blobs: nothing to read.

It’s harmless on five rows; it isn’t on a real table. On a low-cardinality column (region, sex, status), the distribution of blobs can be compared to a known distribution, and frequency analysis often reconstructs the plaintext. Microsoft’s documentation puts it bluntly: an unauthorized user can guess information by examining patterns, especially when the set of possible values is small.

Second weakness: randomized encryption blocks every query. With randomized encryption, the driver adds a fresh random value to each cell before encrypting, so the same input produces a different cyphertext every time. Therefore, the leak is gone but so is the query:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptRand AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptRand = @d
GROUP BY DeptRand;  

Msg 33277, Level 16, State 2, Line 6
Encryption scheme mismatch for columns/variables ‘DeptRand’, ‘@d’. The
encryption scheme for the columns/variables is (encryption_type =
‘RANDOMIZED’, …) and the expression near line ‘6’ expects it to be
DETERMINISTIC, or RANDOMIZED, a BIN2 collation for string data types,
and an enclave-enabled column encryption key, or PLAINTEXT.

The server can no longer compare, since the same plaintext produces a different cyphertext on every row. We gained confidentiality and lost the query. It’s all or nothing.

Third weakness: no range, even with deterministic. Byte equality says nothing about order:

DECLARE @s INT = 55000;
SELECT FirstName FROM dbo.Employees WHERE SalaryDet < @s;

WHERE SalaryDet < @s fails even though the column is deterministic. Sorting, BETWEEN, LIKE: out of reach for Always Encrypted alone.

The verdict is clear: bare Always Encrypted means equality, or nothing (=, IN, GROUP BY, and DISTINCT supported).

MongoDB: queryability lives in a protocol

MongoDB’s Queryable Encryption makes optimal use of randomized encryption. On the server side, everything is Randomized-encrypted, as the documentation explains: the server has no knowledge of the data it processes. The encrypted view (a client connected without the keys) confirms it: even the three Sales employees come out with all-different BinData.

No frequency leak, unlike SQL Server’s deterministic encryption. With the keys, the same find returns the plaintext:

The technical mechanism behind this lies in the collection’s declaration. Each encrypted field carries a queryType, its parameters, and a distinct key (keyId), one Data Encryption Key per field, which is mandatory:

And queries with equality tests, range, and even equality on a field declared as range all work:

On the server side, MongoDB then maintains encrypted index structures, and for each query the driver generates cryptographic tokens that the server checks against those structures without ever seeing the plaintext. This is a searchable encryption scheme. Range is available in GA, with no special hardware.

The real difference comes down to one thing: the driver

On the SQL Server side, the driver’s work stays thin: it encrypts the parameters, rewrites the query, decrypts the results. Queryability itself is already carried by the cyphertext; the driver doesn’t have to handle it. On the MongoDB side, the driver carries the whole protocol: it generates the tokens checked against the encrypted indexes.

Once the collection is in place, find({ department: "Sales" }) is written like a normal query and the driver handles the encryption on its own. Each encrypted field needs its own key (Data Encryption Key). The master key must be pinned, otherwise orphaned keys return an HMAC validation failure error. And the encryption API is only available from a client created as encrypted, so not from a standard Compass connection, for example.

Summary

What if you wanted range while staying on SQL Server?

This is where secure enclaves come in. The engine delegates the computation to an enclave: a protected memory region where the data is decrypted and processed in the clear, out of reach, including from the machine’s administrator. This is what unlocks range, LIKE, sorting, and in-place encryption: inside the enclave the server no longer compares encrypted bytes, it works on the plaintext. This gain has a price. It requires compatible hardware or secure virtualization, an attestation service to deploy and maintain, and keys configured for the enclave, which noticeably increases architectural complexity compared with classic Always Encrypted.

But it also changes the nature of the trust. Queryable Encryption rests on a cryptographic guarantee: the server cannot read, it’s a mathematical property. Enclaves rest on a hardware guarantee: you trust the CPU, and its attestation, to isolate the protected region.

There remains a third path, often mentioned: homomorphic encryption (FHE), which computes directly on the cyphertext without ever decrypting it. Elegant on paper, but out of the game for database search: the computational cost is massive and response time collapses as the volume grows. So the practical choice really does play out between the two worlds described here.

L’article SQL Server vs MongoDB: When the cloud is your adversary (Always Encrypted vs Queryable Encryption) est apparu en premier sur dbi Blog.