The problem

Migrating ~5,600 Oracle tables to SQL Server with SSMA. Most load fine; ~100 tables fail Migrate Data with the same error:

ERROR [42S22] [Oracle][ODBC][Ora]ORA-00904:
"SYS_C00004_21081414:28:22$": invalid identifier

The named column exists in no DDL anyone wrote. The [Ora] prefix says Oracle itself is rejecting the query: SSMA built an extraction SELECT naming a column Oracle refuses to resolve. Tellingly, SELECT * and COUNT(*) run fine against the same table, whatever this column is, Oracle is happy to ignore it, but not to be asked for it by name.

Where the column actually comes from

The name is the giveaway: SYS_C00004_21081414:28:22$ is what Oracle calls a column after ALTER TABLE … SET UNUSED COLUMN.

Oracle offers two ways to get rid of a column: a logical delete and a physical one. The physical delete (ALTER TABLE … DROP COLUMN) is the honest one, but on a large table it is very time- and resource-consuming. That’s why people reach for the logical delete instead:

ALTER TABLE table_name SET UNUSED (column_name);

That statement is metadata-only and instant. The column immediately stops being visible to users, and the physical removal is deferred to whenever there is time for it (see Oracle Documentation):

ALTER TABLE table_name DROP UNUSED COLUMNS;
-- on large tables, cap undo growth by checkpointing every N rows:
ALTER TABLE table_name DROP UNUSED COLUMNS CHECKPOINT 250;

To free the original name for reuse, Oracle renames the column to SYS_C<internal column number>_<YYMMDDHH24:MI:SS>$, sets USER_GENERATED to NO, HIDDEN_COLUMN to YES and releases its COLUMN_ID. So the timestamp is not when the column was added, it is the second someone ran SET UNUSED. Ours says 14 August 2021, 14:28:22.

That also explains the error pattern. The operation is one-way and the column is unreadable by design, so naming it gets you ORA-00904 «invalid identifier». SELECT * and COUNT(*) keep working because the column no longer has a COLUMN_ID and is simply excluded from the star. SSMA, however, lists it and builds an explicit column list Oracle then refuses.

Not to be confused with SYS_NC…$. Those are a different animal: virtual columns backing a function-based index or extended statistics.

Bottom line: returning NULL costs you nothing. This is a column its owner already decided to delete, holding data Oracle itself will no longer hand out. There is no information left to lose.

What doesn’t work for the migration

• SSMA setting Ignore hidden system columns = Yes was not making any effect on this use case
• Dropping the column on SQL Server resolves nothing because the error is on the source SELECT, unaffected.
• Dropping it on Oracle could not be done in our scenario because the source is frozen; DDL not allowed.
• Custom select, column removed or bare NULL: SSMA still expects the name in its mapping and fails with “key not present” or “does not match up” before the query ever reaches Oracle.

Find them all first

Before editing anything, get the full list. Discovering the affected tables one failed migration at a time is a waste of an afternoon because the data dictionary already knows.

The reason the columns are findable at all is an asymmetry between two views: an unused column is gone from ALL_TAB_COLUMNS, but still listed in ALL_TAB_COLS with HIDDEN_COLUMN = 'YES'. That second view is what you query:

SELECT owner,
table_name,
column_name,
data_type,
internal_column_id,
TO_DATE(REGEXP_SUBSTR(column_name, '\d{8}:\d{2}:\d{2}'),
'YYMMDDHH24:MI:SS') AS set_unused_at
FROM dba_tab_cols
WHERE hidden_column = 'YES'
AND user_generated = 'NO'
AND REGEXP_LIKE(column_name, '^SYS_C\d+_\d{8}:\d{2}:\d{2}\$$')
-- AND owner = '<SCHEMA_NAME>'
ORDER BY owner, table_name, internal_column_id;

The regex is deliberately strict: it matches only the SET UNUSED naming pattern, so virtual columns and other system-generated names stay out of the result.

One more view is worth a look, as a cross-check:

SELECT owner, table_name, count AS unused_columns
FROM   dba_unused_col_tabs
--WHERE owner = '<SCHEMA_NAME>'
ORDER  BY count DESC, table_name;

DBA_UNUSED_COL_TABS gives the number of unused columns per table. Sorting by that number puts the dangerous tables first: those with two or three hidden columns are the ones where you’ll forget a line in the custom select and be back at square one.

The fix

Keep the hidden column’s name as an alias, but return a literal NULL instead of reading it. SSMA’s mapping finds the name (no “key not present”); Oracle never resolves the real column (no ORA-00904).

• Tools → Project Settings → General → Migration → enable Extended data migration options.

• Data Migration Settings tab → tick Use custom select → replace each hidden-column line with:

 SELECT ...   
    TO_CHAR("<COLUMN_NAME>", 'TM', 'NLS_NUMERIC_CHARACTERS = ''.,''') as "<COLUMN_NAME>",
    NULL as "SYS_C00004_21081414:28:22$"
from <OWNER>.<TABLE_NAME> t

• Migrate Data → 100%. Drop the NULL-filled column(s) on SQL Server in post-migration cleanup.

Takeaway

SYS_C…$ is not an exotic Oracle feature, it’s an ordinary column someone deleted years ago, logically. Oracle keeps the name on file; SSMA finds it, insists on naming it, and Oracle refuses to hand it over. Aliasing a NULL satisfies both, then you drop the column on the target. No source DDL, no external tooling, everything inside SSMA, behind a project setting that’s hidden by default.

L’article Fixing ORA-00904 on Oracle hidden columns during SSMA data migration est apparu en premier sur dbi Blog.

The starting point: one number from a health check

The I/O statistics of the instance (sys.dm_io_virtual_file_stats) reported the following for tempdb hosted on a local volume D:

And the global view of the volume:

Two details frame the whole investigation:

• Reads are excellent. Writes are 19 times over the threshold. The read path is healthy, the write path is not.
• io_stall_write_ms measures the time between I/O submission and completion, queue time included. A high average does not tell us whether each write is slow or whether writes are waiting behind each other.

When SQL Server performs a write operation, the request goes through the following (simplified) path:

What it is not

The 8 data files show nearly identical latencies and write counts (about 15.8 M each). The tempdb round-robin allocation works perfectly. This is not a hotspot, not a single bad file. The whole volume is affected.

The volume D is local to each cluster node. It is not a shared disk (not on the storage array).

Slow media or queuing?

The DMV cannot separate service time from queue time. So we measured the service time directly at idle with WinSAT:

The media is excellent on all four access profiles. The verdict is simple: the 380 ms are queue time, not service time.

A quick calculation confirms it. The DMV counted about 128 M writes over 104 hours of uptime: about 340 writes per second on average. The volume can absorb 50,000. Average utilization: 0.7%. A volume used at 0.7% that shows 376 ms of average latency means one thing: the load is not smooth. It arrives in bursts. During a burst, thousands of I/Os pile up in the queue, each one waits behind the others and since most of the write volume is concentrated in those bursts they dominate the average.

Where the bursts come from

On this instance, the bursts are produced by sort and hash operations that do not fit in their memory grant and spill to tempdb, mainly during data loads and some heavy analytical queries. The workload side of this story (memory grants, parallelism, NUMA topology) is covered in this blog : https://www.dbi-services.com/blog/wait-stats-and-sub-numa-clustering/

In this post, we follow the storage path only: whatever the workload does, a burst of writes should not cost 380 ms per I/O on a volume this fast.

The invisible layer

When SQL Server writes a page to tempdb, the write goes through this chain:

SQL Server > Windows/NTFS > driver (SmartPqi.sys) > Smart Array controller > physical SSDs

Windows never talks to the SSDs. It talks to the RAID controller (an HPE Smart Array P408i-a) which assembles two SAS SSDs into a RAID 1 mirror and presents the result as volume D.

Reference: https://support.hpe.com/connect/s/softwaredetails?language=en_US&collectionId=MTX-3d51e7d6b8674f16&tab=releaseNotes

Here is the key point: every instrument used so far measures through that controller without seeing it. The DMVs measure above it. WinSAT measures above it. Only one question remains open: how is that card configured? And only one tool answers it: the Smart Storage Administrator CLI (ssacli).

Factory settings

ctrl slot=0 show detail
   
   Cache Board Present: True
   Total Cache Size: 2.0
   Cache Status: Not Configured
   Battery/Capacitor Status: OK
   No-Battery Write Cache: Disabled

ctrl slot=0 ld all show detail
   
   Logical Drive: 2          (volume D)
      Fault Tolerance: 1     (RAID 1)
      Caching: Disabled
      LD Acceleration Method: Smart Path

HPE SSD Smart Path is a direct I/O path: requests skip the RAID firmware stack and go straight to the SSDs. It saves a few dozen microseconds per I/O which benefits reads. But it is a per-volume switch and it is mutually exclusive with the controller cache. Smart Path is enabled by default on every SSD array (factory default). It’s reasonable for a read-oriented volume but it was never revisited for a volume hosting tempdb (one of the most write-intensive profiles there is).

The consequence: every write must be applied to both SSDs of the mirror and confirmed before it is acknowledged. There is no absorber anywhere in the chain. When a burst arrives the queue explodes.

Is enabling the write cache safe?

The old advice “do not enable write caching” targets a different cache: the volatile DRAM inside the disks themselves which loses acknowledged writes on power failure. That one stays disabled (Drive Write Cache Policy: Disable).

The controller cache is a different story. Microsoft’s requirement is stable media: an acknowledged write must survive a power failure. This controller qualifies through the flash-backed write cache mechanism:

• On power loss, the battery does not store any data. It powers the cache module for a few seconds just long enough for the controller to copy the DRAM content to the flash NAND chip on the module itself. Flash is non-volatile: the data survives without any power (indefinitely).
• At reboot the controller restores that data and writes it to the SSDs of the volume before accepting any new I/O.
• If the battery ever fails the controller detects it and automatically falls back to write-through.

There is a second safety belt specific to this volume: tempdb is recreated at every instance startup. Even in the worst theoretical scenario, there is no data anyone would come back for.

The possible fix

Three online reversible commands:

ssacli ctrl slot=0 array B modify ssdsmartpath=disable
ssacli ctrl slot=0 ld 2 modify caching=enable
ssacli ctrl slot=0 modify cacheratio=10/90

Why cacheratio=10/90?

This setting splits the controller cache: 10% for reads, 90% for writes. It is not an exotic choice, it is the HPE factory default for a configured cache documented as the best ratio for most workloads.

Reference: https://support.hpe.com/hpesc/public/docDisplay?docId=a00019059en_us&page=GUID-EE28F5A4-ADF5-4E27-81AA-8377A267FFA7.html

Expected result: LD Acceleration Method: Controller Cache on the logical drive, Cache Status: OK on the controller.

To validate we should not rely on the cumulative DMV averages they will stay polluted by history. We should measure deltas:

• WinSAT after the change: service time should stay excellent (nothing was broken).
• sys.dm_io_virtual_file_stats deltas over a defined window covering the load phases.
• PerfMon during the load window: Avg. Disk sec/Write should stay in single digits at the peak of a burst and the queue should drain between bursts.

One expectation to set correctly: the cache absorbs bursts but it does not add throughput. All the bytes still land on the same two SSDs.

Note: these commands have not been implemented. They are proposals only. The change must be reviewed, validated and scheduled by the customer before any implementation.

Local tempdb volumes on FCI nodes: a good idea

Placing tempdb on a local volume in a Failover Cluster Instance is supported since SQL Server 2012 and it is a good design: tempdb is recreated at startup so there is nothing to fail over. It offloads the shared storage and local SSDs deliver excellent performance for one of the hottest write profiles of the instance (our measurements above prove it).

But this choice turns storage health into a per-node responsibility:

• Check the RAID controller configuration on every node. The passive node most likely carries the same factory default. After a failover the problem would silently come back.
• Monitor Battery/Capacitor Status. A dead battery silently disables the write cache and brings the symptom back.

The architecture is right. It just makes your RAID controller part of your database health check.

Thank you. Amine Haloui

L’article When tempdb write latency points below SQL Server est apparu en premier sur dbi Blog.

When a company still refuses to put its sensitive data in the cloud, the reason usually isn’t cost or performance: it’s data sovereignty. In the cloud, someone else is the administrator of the machine; therefore the provider can, in theory, read the data files, see the data being used in memory, or read the backups. The threat model is no longer the external attacker but the privileged insider hosting the database.

As covered previously on my blog Beyond TDE and TLS: Bridging the Data Security Governance Gap in Lower Environments, various encryption methods can protect you. For example, TLS protects data in transit and TDE protects it at rest, but as soon as the engine runs a query, it handles plaintext in memory. So there are three states to protect: at-rest, in-transit, in-use. The gap this article focuses on is the last one.

That is exactly what Always Encrypted (SQL Server) and Queryable Encryption (MongoDB) target. The common principle: encryption and decryption happen client-side, in the driver; the keys never reach the engine. The data stays encrypted at rest, in transit, and during processing. The DBA, the cloud operator, the hypervisor admin: all of them see only cyphertext.

That leaves one question: if the engine sees only cyphertext, how does it answer a WHERE condition? Both database engines do answer it, but through different technical means.

SQL Server: queryability lives in the cyphertext

To demonstrate all this, let’s start by creating a table with two columns, Salary and Department, encrypted deterministically on one side and randomized on the other:

DROP TABLE IF EXISTS dbo.Employees;
CREATE TABLE dbo.Employees (
  Id         INT IDENTITY(1,1) PRIMARY KEY,
  LastName   NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  FirstName  NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  DeptDet    NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be DETERMINISTIC
  DeptRand   NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be RANDOMIZED
  SalaryDet  INT NOT NULL, -- will be DETERMINISTIC
  SalaryRand INT NOT NULL  -- will be RANDOMIZED
);
GO

INSERT INTO dbo.Employees (LastName, FirstName, DeptDet, DeptRand, SalaryDet, SalaryRand) VALUES
('Martin',  'Alice', 'Sales', 'Sales', 55000, 55000),
('Dubois',  'Bob',   'Sales', 'Sales', 48000, 48000),
('Bernard', 'Chloe', 'Sales', 'Sales', 52000, 52000),
('Petit',   'David', 'IT',    'IT',    72000, 72000),
('Durand',  'Emma',  'IT',    'IT',    68000, 68000);

Five rows: three Sales, two IT. The BIN2 collation is required by Always Encrypted (link to documentation), and the master key lives outside the database (Key Vault, certificate store, or HSM).

Column encryption isn’t done in T-SQL, because the engine doesn’t have the keys. It’s driven from the client (here in PowerShell), declaring for each value a deterministic column and its randomized twin:

Import-Module SqlServer -MinimumVersion 22.0.59
$sqlConnectionString = "Data Source=.\LAB2025;Initial Catalog=AEDEMO;Integrated Security=True;Encrypt=False;Trust Server Certificate=False"
$smoDatabase = Get-SqlDatabase -ConnectionString $sqlConnectionString

$encryptionChanges  = @()
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptDet    -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptRand   -EncryptionType Randomized    -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryDet  -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryRand -EncryptionType Randomized    -EncryptionKey "CEK1"

Set-SqlColumnEncryption -ColumnEncryptionSettings $encryptionChanges -InputObject $smoDatabase

In this example, I’m working on my 2025 SQL Server instance, on the AEDEMO database, using the column encryption key CEK1 I created beforehand (itself protected by a column master key stored outside the database).

We check that the engine sees the right type per column:

SELECT c.name, c.encryption_type_desc
FROM sys.columns c
WHERE c.object_id = OBJECT_ID('dbo.Employees');

The deterministic mechanism works like this: same plaintext, same cyphertext (an injective function). On an Always Encrypted-enabled connection with Parameterization for Always Encrypted and parameters for the predicates (never literals), equality works:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptDet AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptDet = @d
GROUP BY DeptDet;

The driver encrypts @d with the same key, the server finds the encrypted values that are identical to the parameter, and the result is correct.

You pay for it in three ways…

First weakness: deterministic encryption can cause a data leak. You don’t need the keys to see it. Just connect without Always Encrypted and read the table: the encrypted columns come out as raw binary.

SELECT FirstName, DeptDet, DeptRand FROM dbo.Employees;

Look at the DeptDet column: Alice, Bob, and Chloe share exactly the same blob, and David and Emma share another. Two distinct values across five rows. The adversary doesn’t know what 0x012536… means, but reads the structure: two departments, one with three people, the other with two, and who goes with whom. The DeptRand column, on the other hand, shows five all-different blobs: nothing to read.

It’s harmless on five rows; it isn’t on a real table. On a low-cardinality column (region, sex, status), the distribution of blobs can be compared to a known distribution, and frequency analysis often reconstructs the plaintext. Microsoft’s documentation puts it bluntly: an unauthorized user can guess information by examining patterns, especially when the set of possible values is small.

Second weakness: randomized encryption blocks every query. With randomized encryption, the driver adds a fresh random value to each cell before encrypting, so the same input produces a different cyphertext every time. Therefore, the leak is gone but so is the query:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptRand AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptRand = @d
GROUP BY DeptRand;  

Msg 33277, Level 16, State 2, Line 6
Encryption scheme mismatch for columns/variables ‘DeptRand’, ‘@d’. The
encryption scheme for the columns/variables is (encryption_type =
‘RANDOMIZED’, …) and the expression near line ‘6’ expects it to be
DETERMINISTIC, or RANDOMIZED, a BIN2 collation for string data types,
and an enclave-enabled column encryption key, or PLAINTEXT.

The server can no longer compare, since the same plaintext produces a different cyphertext on every row. We gained confidentiality and lost the query. It’s all or nothing.

Third weakness: no range, even with deterministic. Byte equality says nothing about order:

DECLARE @s INT = 55000;
SELECT FirstName FROM dbo.Employees WHERE SalaryDet < @s;

WHERE SalaryDet < @s fails even though the column is deterministic. Sorting, BETWEEN, LIKE: out of reach for Always Encrypted alone.

The verdict is clear: bare Always Encrypted means equality, or nothing (=, IN, GROUP BY, and DISTINCT supported).

MongoDB: queryability lives in a protocol

MongoDB’s Queryable Encryption makes optimal use of randomized encryption. On the server side, everything is Randomized-encrypted, as the documentation explains: the server has no knowledge of the data it processes. The encrypted view (a client connected without the keys) confirms it: even the three Sales employees come out with all-different BinData.

No frequency leak, unlike SQL Server’s deterministic encryption. With the keys, the same find returns the plaintext:

The technical mechanism behind this lies in the collection’s declaration. Each encrypted field carries a queryType, its parameters, and a distinct key (keyId), one Data Encryption Key per field, which is mandatory:

And queries with equality tests, range, and even equality on a field declared as range all work:

On the server side, MongoDB then maintains encrypted index structures, and for each query the driver generates cryptographic tokens that the server checks against those structures without ever seeing the plaintext. This is a searchable encryption scheme. Range is available in GA, with no special hardware.

The real difference comes down to one thing: the driver

On the SQL Server side, the driver’s work stays thin: it encrypts the parameters, rewrites the query, decrypts the results. Queryability itself is already carried by the cyphertext; the driver doesn’t have to handle it. On the MongoDB side, the driver carries the whole protocol: it generates the tokens checked against the encrypted indexes.

Once the collection is in place, find({ department: "Sales" }) is written like a normal query and the driver handles the encryption on its own. Each encrypted field needs its own key (Data Encryption Key). The master key must be pinned, otherwise orphaned keys return an HMAC validation failure error. And the encryption API is only available from a client created as encrypted, so not from a standard Compass connection, for example.

Summary

What if you wanted range while staying on SQL Server?

This is where secure enclaves come in. The engine delegates the computation to an enclave: a protected memory region where the data is decrypted and processed in the clear, out of reach, including from the machine’s administrator. This is what unlocks range, LIKE, sorting, and in-place encryption: inside the enclave the server no longer compares encrypted bytes, it works on the plaintext. This gain has a price. It requires compatible hardware or secure virtualization, an attestation service to deploy and maintain, and keys configured for the enclave, which noticeably increases architectural complexity compared with classic Always Encrypted.

But it also changes the nature of the trust. Queryable Encryption rests on a cryptographic guarantee: the server cannot read, it’s a mathematical property. Enclaves rest on a hardware guarantee: you trust the CPU, and its attestation, to isolate the protected region.

There remains a third path, often mentioned: homomorphic encryption (FHE), which computes directly on the cyphertext without ever decrypting it. Elegant on paper, but out of the game for database search: the computational cost is massive and response time collapses as the volume grows. So the practical choice really does play out between the two worlds described here.

L’article SQL Server vs MongoDB: When the cloud is your adversary (Always Encrypted vs Queryable Encryption) est apparu en premier sur dbi Blog.

Here are the questions that arise 

• Can a wait type really account for more than 100% of server uptime? 
• Are these values a problem in themselves? 
• What do they tell us about the configuration of the instance? 

The waits themselves were not the problem. But they pointed us to a BIOS option (Sub-NUMA Clustering) that was silently reshaping the entire NUMA topology of the server and putting it in conflict with the MAXDOP configuration. 

This post is the story of that investigation: from the wait statistics to the BIOS, step by step. 

Unusual wait statistics 

The collected data 

The instance had been up for about 11 days (roughly 273 hours). The two top wait types were: 

CXPACKET and CXCONSUMER 

Both wait types belong to parallel query execution. A parallel plan splits its work into branches and the branches synchronize at exchange operators. 

• CXCONSUMER: the consumer side of an exchange is simply waiting for the producers to deliver rows. This is the structural, unavoidable wait of any parallel plan. It is considered benign. 

• CXPACKET: a thread has finished its share of the work and waits for slower branches at the synchronization point. This is the potentially actionable signal: it reveals an imbalance between the branches of the plan. 

An analogy: in a factory, CXCONSUMER is a workstation waiting for parts to arrive (normal), CXPACKET is a workstation that has finished its batch and waits for a slower colleague (an imbalance worth examining). 

Are these values abnormal, then? 

Look at the averages again: 0.00 ms and 1.00 ms per wait. 657.78 hours at 1 ms average means roughly 2.4 billion individual waits. These counters do not describe long blockings, they describe the normal tick-tock of exchange operators in a heavily parallel workload. A server suffering from severe skew would show averages of tens of milliseconds. 

The healthcheck tool itself says it in its own message: “Usually a cost threshold for parallelism / MAXDOP tuning issue rather than a problem in itself.” The line is an inventory entry (any wait above 10% of uptime gets reported), not an alarm. 

So why did these two lines matter? Not because of their height because of their rank. CXCONSUMER and CXPACKET were number 1 and number 2 of the entire wait inventory, ahead of everything else. Their rank designated parallelism as the dominant workload of this instance and therefore its configuration as the first thing to confront with the topology. 

The instance configuration 

SELECT [name], value_in_use 
FROM sys.configurations 
WHERE [name] IN (N'max degree of parallelism', N'cost threshold for parallelism'); 

• Cost threshold for parallelism: 50, correctly raised from the default of 5. This setting decides which queries are allowed to go parallel (those whose estimated cost exceeds the threshold). 
• MAXDOP: 8, this setting decides how wide: the maximum number of worker threads per parallel branch. 

MAXDOP 8 is a perfectly reasonable value in isolation. The problem only appears when we look at the topology. 

The topology 

SELECT cpu_count, hyperthread_ratio, socket_count, cores_per_socket, 
numa_node_count, softnuma_configuration_desc 
FROM sys.dm_os_sys_info; 

Two sockets. Four NUMA nodes. That is the anomaly this whole post is about. 

From the anomaly to the root cause 

On modern processors, there is no central memory controller shared by all cores. Each processor package has its own memory controllers with its own RAM modules attached to them. Such a group (cores + memory controllers + local RAM) is a NUMA node. 

When a core reads an address that lives in its own node’s RAM: direct path, fast. When it reads an address that lives in another node’s RAM: the request crosses the interconnect, with roughly 1.5 to 2 times the latency. That is the meaning of the name: Non-Uniform Memory Access. The cost of a memory access depends on the physical distance between the core that asks and the RAM module that answers. 

The natural NUMA boundary is therefore the socket: one socket = its memory controllers = its local RAM = one NUMA node. 

Here we have 4 nodes for 2 sockets: 2 nodes per socket. Only three mechanisms can produce that: 

Cross-checking with the hardware 

The server is a physical HPE ProLiant DL380 Gen10 with 2 × Intel Xeon Gold 6244 (8 cores / 16 threads each, 3.60 GHz base) and 384 GB of RAM (12 × 32 GB, balanced across the memory channels). 

Windows server shows “8 Core(s), 8 Logical Processor(s)” per socket, and Task Manager shows Cores: 16 = Logical processors: 16. Hyper-threading is disabled: 16 physical cores, one logical processor per core. This matters for the rest of the post, because Microsoft’s MAXDOP recommendations are expressed in logical processors per node. 

A corroborating detail: Task Manager reports L3 cache = 99.0 MB. The Gold 6244 has 24.75 MB of L3 per socket, so the server has 49.5 MB but 99.0 = 4 × 24.75. Windows counts the socket’s L3 once per NUMA node it is presented with. The only wrong line in the cache arithmetic is exactly the one that depends on NUMA counting. 

What Sub-NUMA Clustering is 

Our socket contains 8 cores, 6 memory channels and a shared L3 cache connected by an internal mesh. Sub-NUMA Clustering (SNC) is a BIOS option that cuts this socket into two domains, each with 4 cores, 3 memory channels and its half of the L3 affinity and presents each half as a full NUMA node to the operating system. 

Two sockets × SNC = 4 NUMA nodes of 4 logical processors and about 96 GB of RAM each. Exactly what our DMVs show. 

Why does this option exist? Because for some workloads it helps. A public SPEC CPU2017 result published by Dell on the equivalent platform (PowerEdge R640, same 2 × Xeon Gold 6244, same 384 GB in 12 × 32 GB) is instructive on this point the BIOS notes literally list “Sub NUMA Cluster enabled” and the published numactl output shows the resulting topology: 

Reference : https://www.spec.org/cpu2017/results/res2019q2/cpu2017-20190429-12798.html

What SNC makes SQLOS build 

At startup, SQLOS mirrors the presented topology: one memory node and one group of schedulers per NUMA node. On this server: 4 groups of 4 schedulers and the decisive point the buffer pool is partitioned across the 4 nodes. With max server memory at 210 GB, each node manages roughly 52 GB, and a cached data page physically lives in the RAM of one node. 

The mechanics then the corrections 

Now we can close the loop with Part 1: 

• SNC (BIOS) cuts 2 sockets into 4 NUMA nodes of 4 logical processors. 

• MAXDOP is 8 and a node only offers 4 schedulers: every parallel query spans two nodes by construction at every execution. 

• The workers on the remote node access non-local memory. This cost is mostly invisible in the wait statistics a thread reading remote memory is RUNNING, not waiting. The direct cost hides in queries that simply run slower. 

• Only the indirect effect surfaces: asymmetric memory distances desynchronize the branches, the fast workers finish their packets and wait for the slow ones at the exchanges and that spills into CXPACKET. 

• The placement follows the load of the moment, so the same query can have different costs from one execution to the next. SNC does not only add cost it adds variance. 

Why MAXDOP 4 alone is not the fix 

Capping MAXDOP at 4 confines each query’s workers to a single node. That offers two things: the working memory (memory grants, hash tables, sort runs, exchange buffers) becomes local and all branches advance at the same speed and nobody waits for a remote colleague anymore. 

But it does not buy data locality. A buffer pool page is allocated on the node of the worker that read it from disk  potentially days ago for another query on another node and it never migrates afterwards. With 4 nodes, a query confined to node 2 finds on average only about 25% of the already-cached pages locally. And MAXDOP 4 also halves the width of every query on a data warehouse that lives on parallelism. MAXDOP 4 is the bandage. The correction is the topology itself. 

Microsoft’s recommendations 

Our server sits on the third line in every scenario: 

Current (SNC enabled) : 

After (with SNC disabled) : 

The possible two corrections: 

Short term (online, reversible, no restart): MAXDOP = 4. It confines each query to one node as the server is presented today. It is a transitional measure not the target state. 

EXEC sp_configure 'max degree of parallelism', 4; 
RECONFIGURE;

Root-cause correction (through a maintenance window): disable SNC in the BIOS. On an HPE Gen10, the option lives in: 

System Configuration > BIOS/Platform Configuration (RBSU) > Power and Performance Options > Sub-NUMA Clustering > Disabled 

It looks like that: 

Reference : https://lenovopress.lenovo.com/lp1499.pdf

Conclusion

The wait statistics never proved anything in this story and that is the point. Values above 100% of uptime are normal (they cumulate across all simultaneous waiters), the averages were small and CXCONSUMER is benign by nature. What the two lines provided was a characterization: normalized by uptime, they showed about 2.5 threads permanently synchronizing exchanges and this instance lives on parallelism. Their rank designated parallelism as the dominant workload and therefore its configuration as the first thing to confront with the topology. 

Thank you. Amine Haloui

L’article Wait stats and Sub-NUMA clustering  est apparu en premier sur dbi Blog.

This is something we often see among our customers. If we assume that we carry out a migration project roughly every five years, this means we’re keeping databases that are taking up space for no reason.

We, however, are Green SQL Server DBA; we’ll look to manage it correctly.

The AUDIT

First of all, what do we need to analyse?

This does not apply to all databases, but only to those that are obsolete.

They can be categorised into two types:

• Databases Never Accessed
  • No application connections detected
  • No recent transactions or queries after a period (3 months for exemple)
• Abandoned Databases
  • Linked to cancelled projects
  • No activity and nobody tell it

To see databases never accessed and abandoned databases I use the DMV sys.dm_db_index_usage_stats to see if I have an access to the database (read or write):

SELECT d.name AS DatabaseName, MAX(ius.last_user_seek) AS LastSeek,MAX(ius.last_user_scan) AS LastScan,
    MAX(ius.last_user_lookup) AS LastLookup,

    (
        SELECT MAX(v.ActivityDate)
        FROM (VALUES
                (MAX(ius.last_user_seek)),
                (MAX(ius.last_user_scan)),
                (MAX(ius.last_user_lookup))
             ) v(ActivityDate)
    ) AS LastReadActivity,

    MAX(ius.last_user_update) AS LastWriteActivity,

    DATEDIFF(DAY,
        (
            SELECT MAX(v.ActivityDate)
            FROM (VALUES
                    (MAX(ius.last_user_seek)),
                    (MAX(ius.last_user_scan)),
                    (MAX(ius.last_user_lookup))
                 ) v(ActivityDate)
        ),
        GETDATE()
    ) AS DaysSinceLastRead,

    DATEDIFF(DAY,
        MAX(ius.last_user_update),
        GETDATE()
    ) AS DaysSinceLastWrite

FROM sys.databases d
LEFT JOIN sys.dm_db_index_usage_stats ius
    ON d.database_id = ius.database_id
WHERE d.database_id > 4  -- Exclude system databases
GROUP BY d.name
ORDER BY LastWriteActivity ASC;

We can see on the first databases in the list, that we have null everywhere… This means that the databases were not solicited since the last restart… Good candidate for unused databases!

This brings us to the next point.

The ANALISYS

If we can see that there is little activity on the database, we can, of course, carry out an initial analysis using criteria such as these:

< 30 days –> the database is still active

30–180 days –> the database is on hold pending to be dropped

> 180 days –> the database is a strong candidate to be dropped

Of course, it also depends on your environment – whether it’s dev, test or prod – and the criteria will vary accordingly.

The IMPACT

Like in my precedent post, we will have same impacts but not for queries of courses!

The first impact is the storage and the backup

If the database represents ~100 GB on the disk , it’s also 100 GB backed up unnecessarily.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 1 TB!

The third impact is on the maintenance plan

The following operations must also handle the unused database:

– CheckDB
– Backups

As the database is not used, the index rebuild/reorg & update Stats will be fast!

Each unused database extends the maintenance windows for the checkdb and backups…

The ADVISE

The first advice would be to plan well in advance.

Indeed, when creating or restoring a database, it is important to know its lifecycle.

Don’t hesitate to challenge the application owners on a deadline and make sure to add it to your calendar +1day.

The second advice is to set up a monthly monitoring process prior to Windows or SQL Server patch cycles so that you can identify these unnecessary databases.

And before dropping a database, I advice you to put the database offline and see if after some days somebody complaints…

In the majority of cases, you will not have reactions, believe me!

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will let the unused databases until the next migration.

A Green SQL Server DBA will check it periodically and manage it.

An unused database is like a car that never leaves the garage.
 It still costs money, requires maintenance, and consumes resources.
The Green SQL Server DBA notices it before it becomes waste…

Think about it and begin now to see to monitor your unused databases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #3 – The hidden cost of forgotten databases est apparu en premier sur dbi Blog.

After creating the Ticket, I connect to the instance and see that the msdb system database is in a Suspect State:

After taking a deep breath because I don’t see that every day, I begin my investigation by asking the state with the system view sys.databases first:

msdb is really in a suspect mode…

Now, to begin the analysis, I go to read the error log.:

The Error Log give me the error that I comments in SSMS query to read it better:

Error: 3314, Severity: 21, State: 1.
During undoing of a logged operation in database ‘msdb’, an error occurred at log record ID (821931:128397:5).
Typically, the specific failure is logged previously as an error in the Windows Event Log service. Restore the database or file from a backup, or repair the database.

If you have this error message, the best is to restore the msdb;
As the information of the backups is in the msdb, I cannot do a right-click to restore the database…  

I looked for the latest backup in the error log and find it:

But I also sse that the checkdb before each backup give 12 errors…

My question is at this point: Wasn’t there an alert saying that the database was corrupted?

As the users are blocked, I need to react quickly.
This is why, I do my restore and a CHECKDB with  REPAIR_ALLOW_DATA_LOSS:

RESTORE DATABASE msdb FROM DISK = 'xxx\msdb\msdb_fullbackup_xxxx20260801xxxx.BAK' WITH REPLACE;
GO
 ALTER DATABASE msdb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
GO
DBCC CHECKDB ('msdb', REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS, ALL_ERRORMSGS;
GO
ALTER DATABASE msdb SET MULTI_USER;

The commands completed successfully! YES!

The database msdb is online and working fine. SUPER!

To be sure, I go in the errorlog to see the restore and the checkdb with REPAIR_ALLOW_DATA_LOSS:

The restore was successful and in the checkDB with can see 12 errors and repaired 12 errors.

If I will investigate deeper, I will go the dump file and analyse it.

I finally managed to sort out and resolve the critical alert on monday morning.
Users can connect and read their tables again…
The customer knows why he comes to us for an SLA…

A good victory for a monday morning in our support SLA dbi-services!
See you soon for sharing again customer cases!

L’article SQL Server:  msdb is in Suspect State est apparu en premier sur dbi Blog.

We turn up on a Monday morning and there are alerts from the weekend saying

[MSSQL] – Instance Dark_Vador-Data(F:): Disk space in GB on server Death_Star is critical

Info: NOT OK – Free space: 20 MB out of 200 GB

The storage is almost full and:
– no projects were deployed over the weekend
– no exceptional data loads were scheduled
– nobody works at the weekend

A strange phenomenon to come across on a Monday morning….

On closer inspection, the findings are surprising; we can see that a database within the instance has been undergoing regular autogrowth for weeks…

Every day, there are several regular autogrowth on the data file for this database until the saturation of the disk.

Who is the culprit?

Not the business data, but one or more tables named xxxxLog or Historyxxx, or the combination of both HistoryLog!

PS: It’s an example but in the reality before the weekend, we will have Warnings and in the majority of the cases we react before the critical alert of course!

Data created to diagnose a problem, track an activity or maintain a history, which is then never cleaned up or has a default retention period that is too long and unsuitable for the context, is not always easy to detect.

On the other hand, it’s always easier to expand disk capacity without too much effort, and everything runs smoothly in an ideal world…

We, however, are Green SQL Server DBA; we’ll look deeper.
Our curiosity will lead us to challenge the business on these points.

The AUDIT

First, you need to know the autogrowth settings for these databases and how the data file growth.
Here is a simple script that shows the settings:

SELECT
    DB_NAME(database_id) AS DatabaseName,
    name AS LogicalFileName,
    type_desc AS FileType,
    physical_name AS PhysicalFileName,
    size / 128.0 AS CurrentSizeMB,
    CASE
        WHEN is_percent_growth = 1
            THEN CAST(growth AS VARCHAR(10)) + ' %'
        ELSE CAST(growth / 128 AS VARCHAR(10)) + ' MB'
    END AS AutoGrowth,
    max_size,
    CASE
        WHEN max_size = -1 THEN 'Unlimited'
        ELSE CAST(max_size / 128 AS VARCHAR(20)) + ' MB'
    END AS MaxSize
FROM sys.master_files where type_desc='ROWS' AND database_id > 4
ORDER BY
    DatabaseName,
    FileType,
    LogicalFileName;

Like for my precedent blog “Green SQL Server DBA Tips  #1 – Are unnecessary indexes cluttering up your database?”, I use the Dynamics database as example:

In my script, I exclude all system databases and the Log File to be concentrated to the Data File.

Now, I need to have the data file growth. This is through the default trace file and the event 92:

DECLARE @TraceFile NVARCHAR(500);

SELECT @TraceFile = path
FROM sys.traces
WHERE is_default = 1;

IF @TraceFile IS NULL
BEGIN
    RAISERROR('Default Trace is disabled.',16,1);
    RETURN;
END

;WITH GrowthEvents AS
(
    SELECT
        DB_NAME(DatabaseID) AS DatabaseName, FileName, CAST(StartTime AS DATE) AS GrowthDate,
        DATEPART(YEAR, StartTime) AS YearNum, DATEPART(WEEK, StartTime) AS WeekNum,
        DATEPART(MONTH, StartTime) AS MonthNum, IntegerData * 8.0 / 1024 AS GrowthMB
    FROM fn_trace_gettable(@TraceFile, DEFAULT)
    WHERE EventClass IN (92) -- 92 is data growth
)

SELECT DatabaseName,FileName,GrowthDate,COUNT(*) AS GrowthEvents,SUM(GrowthMB) AS TotalGrowthMB
FROM GrowthEvents GROUP BY DatabaseName,FileName,GrowthDate ORDER BY DatabaseName, GrowthDate

As I don’t have any growth on the dynamicsBC database, I switch to the M-Files database for my example:

We have here some daily growth…

Go to see if we have historical or log tables in M-Files with this simple script who give us  the table and the row count and MB:

SELECT s.name AS Schema_Name,t.name AS Table_Name,
    SUM(p.rows) AS Row_Count, CAST(SUM(a.total_pages) * 8.0 / 1024 AS DECIMAL(18,2)) AS SizeMB
FROM sys.tables t INNER JOIN sys.schemas s     ON t.schema_id = s.schema_id
INNER JOIN sys.indexes i ON t.object_id = i.object_id
INNER JOIN sys.partitions p ON i.object_id = p.object_id  AND i.index_id = p.index_id
INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id
WHERE t.name LIKE 'Histor%' OR t.name LIKE '%Log'
GROUP BY s.name, t.name ORDER BY SizeMB DESC;

The ANALISYS

In my case, we can see that we have 383550 lignes (19.73Mb) on the table named OBJECTTYPECHANGELOG for a M-Files database.

It’s only ~20MB here but in few months perhaps 20 GB or more…

As a Green SQL Server DBA, I will contact the owner of M-Files project otherwise don’t forget that dbi services has also the expertise in ECM tools like M-Files.

The IMPACT

Like in my precedent post, we will have the same impacts

The first impact is on DML INSERT operations.

With every insertion, the log or historical information is written to the table and update perhaps also index and at the end is never used by a query.

The second impact is the storage and the backup

If the log table represents ~20 GB of the database.

It’s 20 GB backed up unnecessarily, 20 GB restored unnecessarily and, above all, 20 GB stored unnecessarily and just for one environment.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 100 GB!

The third impact is on the maintenance plan

The following operations must also handle the log/historical tables:
– Rebuild Index
– Reorganise Index
– CheckDB
– Backups
– Restores

Each growth of log/historical tables extends the maintenance windows…

The ADVISE

You really need to be careful before deleting anything in these table.

Don’t Truncate the table because you see it!

First see with the vendor if a purge or a retention parameter exists through the application. 

In my case with M-Files and the table OBJECTTYPECHANGELOG, in the administration interface, you can configure the retention. Go to challenge your M-Files team on it!

Finally, for every large log or historical table, you need to ask yourself the following questions:
– Who uses it?
– How often?
– Is there a legal obligation?
– What retention period is required?
– Can it be archived?
– Can it be purged?

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will see a data disk full and says:

“How much GB need we to extend the disk?”

A Green SQL Server DBA sees a data disk full and says:

“Why is my disk full and what can I do?”

Just like adding a larger trunk doesn’t make a car more efficient, adding more GB doesn’t make a database healthier.

The greenest SQL Server DBA is not the one with the largest disks but it’s the one that keeps only data that continues to deliver value.

Think about it and begin now to see if you have cases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #2 – When logs and history records cause your database to growth and crash est apparu en premier sur dbi Blog.

Logical fragmentation was a problem on spinning disks: an ordered scan on a fragmented index broke the read-ahead into smaller I/Os and each jump cost a disk head movement. On SSD, NVMe and cloud storage, the penalty of non-sequential reads is marginal. And the “physical order” we used to restore so carefully is only the order of the pages inside the data file: below it, the file system, the SSD controller and the storage layer place the blocks wherever they want anyway.

Here is the mapping to remember

• Internal fragmentation = page density = fixed by automatic index compaction
• External fragmentation = page order = ignored by automatic index compaction (it can even increase and we will see it in the demo)

How does it work ?

Automatic index compaction is not a hidden maintenance job. It is an additional task performed by a background process that already exists: the PVS cleaner is a component of Accelerated Database Recovery (ADR).

Here is the principle

• The PVS cleaner periodically visits the pages that were recently modified (insert, update, delete) to remove obsolete row versions
• When automatic index compaction is enabled the cleaner also checks if the visited page has free space excluding the space reserved by the fill factor
• If so, it moves rows from the next page into the current page as long as they fit and repeats the operation on a few consecutive page pairs
• A page that becomes empty is deallocated

The result: the number of pages decreases, the page density increases, and the storage space, I/O, CPU and buffer pool consumption decrease.

The overhead is minimal because the process only considers recently modified pages, unlike a rebuild or a reorganize which process all the pages. Like a reorganize, the compaction acquires short-term exclusive page locks to move the rows. If a lock cannot be acquired immediately, the page is simply skipped and will be considered again later.

One command per database, no restart, no exclusive access. The compaction starts within minutes:

-- Enable the feature
ALTER DATABASE [SQL-DB-1] SET AUTOMATIC_INDEX_COMPACTION = ON;

-- Check if it's enabled
SELECT name, is_automatic_index_compaction_on FROM sys.databases;

One important point before you enable it

The compaction process only considers the pages modified after you enable the feature. If the page density of your indexes is already low, run a one-time reorganize or rebuild to fix the existing situation. From that point on, the automatic compaction keeps the indexes compact without any action on your side.

Demo

• We create a table with a clustered primary key and we insert 5 million rows
• We measure the baseline: page count, page density, fragmentation
• We delete 2 rows out of 3, scattered over the whole table, to simulate index bloat
• We measure again and we let the engine work

We create the table and we insert 50K rows:

CREATE TABLE dbo.Demo (
    Id      int IDENTITY CONSTRAINT PK_Demo PRIMARY KEY CLUSTERED,
    Payload char(400) NOT NULL DEFAULT REPLICATE('X', 400)
);

INSERT INTO dbo.Demo (Payload)
SELECT TOP (50000) REPLICATE('X', 400)
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

We measure the baseline:

SELECT index_level, page_count, avg_page_space_used_in_percent,
       avg_fragmentation_in_percent, record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Demo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

We measure the baseline:

SELECT index_level, page_count, avg_page_space_used_in_percent,
       avg_fragmentation_in_percent, record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Demo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

The internal fragmentation is low.

Here is what we have:

However to observe the automatic index compaction in action we need to create the following situation:

Here is the baseline:

We create the bloat

We delete 2 rows out of 3 scattered uniformly over the table. No page becomes empty. Every page keeps around one third of its rows. We change page density:

DELETE FROM dbo.Demo WHERE Id % 3 <> 0;

Without automatic index compaction, the expectation is simple: the page count stays at 277778 (deletes alone do not deallocate non-empty pages) and the density drops to around 31% (one third of 94.93%).

We measure again

We let the engine finish its job and we check again

The engine did the work in the background while the database stayed fully available.

This feature replaces the compaction part of your maintenance strategy not the whole strategy.

A summary:

• It does not update the statistics. A rebuild updates them the automatic compaction does not. If you rely on your rebuild jobs to refresh the statistics, keep a statistics update job.
• It does not reduce the logical fragmentation and it can even increase as we saw in the demo. For most workloads this has no measurable impact
• It does not recreate the free space reserved by the fill factor. Only a rebuild does. Workloads that need a fill factor below 100 to reduce page splits can still benefit from an occasional rebuild
• It does not shrink the data files. The used space decreases, the allocated size does not change
• It only applies to the leaf level of B-tree indexes in IN_ROW_DATA allocation units. Heaps, LOB data, row-overflow data, compressed columnstore rowgroups and memory-optimized tables are not concerned
• Indexes with page locks disabled (ALLOW_PAGE_LOCKS = OFF) are not eligible

Reference

Automatic Index Compaction – SQL Server | Microsoft Learn

Thank you. Amine Haloui

L’article SQL Server: Automatic index compaction in Azure (preview) – Part 2 est apparu en premier sur dbi Blog.

Microsoft recently released a new feature in public preview: automatic index compaction. The database engine takes care of compacting the indexes itself, in the background, while the data changes. No job, no maintenance window, no script to maintain.

Here is what we will cover in this blog post

• Which problems this feature addresses
• The requirements
• The difference between page density and index fragmentation (and why it matters)
• What the feature does not do

This is the first part of a series of two blog posts.

Which problems does it address?

Automatic index compaction is Microsoft’s answer to a routine operational task: index maintenance jobs.

Here are the problems with these jobs

• They are expensive in CPU and I/O. A rebuild reads and rewrites the entire index
• They take time to run, and maintenance windows keep shrinking
• They generate a lot of transaction log, which impacts log backups and Always On replication
• They can block applications. An offline rebuild locks the table and even an online rebuild needs a short exclusive lock at the beginning and at the end
• Someone has to set them up, monitor them and fix them when they fail

There is also a less obvious problem: for years, these jobs have been optimizing the wrong metric. We will come back to this point in a moment.

Requirements

The feature is currently in public preview and available on

• Azure SQL Database
• Azure SQL Managed Instance, with the Always-up-to-date update policy
• SQL database in Microsoft Fabric

It is not available on SQL Server on-premises, including SQL Server 2025.

There is nothing to install. The feature relies on a background process that already runs on these platforms (the ADR cleaner, more on this below), and Accelerated Database Recovery is always enabled on Azure SQL Database, Azure SQL Managed Instance and SQL database in Fabric.

Internal and external fragmentation

When we talk about index fragmentation, we are actually talking about two different phenomena. They are measured by two different columns of sys.dm_db_index_physical_stats, they have different causes, and above all they have very different costs.

Before we start: how is a data file organized ?

To understand fragmentation, we need one prerequisite: how SQL Server sees its data files.

• For any program, a file is an array of bytes provided by the operating system. Byte 0, byte 1, byte 2, and so on. Reading means asking for a range: an offset and a length

• SQL Server slices this array into pages of 8 KB. The page N is, by definition, the bytes N × 8192 to (N+1) × 8192 − 1. The page ID is not a label, it is an address in the file

• Everything below the file (NTFS clusters, SSD, SAN volumes) is translated and hidden by the storage stack.

Every time we say “physical order” in the rest of this post, we mean the order of the pages inside the data file not their placement on the hardware.

What does it look like?

Internal fragmentation: the pages are not full

Internal fragmentation means that the pages of an index contain free space. It is measured by avg_page_space_used_in_percent, also called page density.

Where does it come from?

• Deletes, especially scattered deletes: the rows disappear, the pages stay
• Page splits: when a page is full, an insert in the middle of the key range or an update that makes a row grow moves half of the rows to a newly allocated page. Both pages end up around 50% full
• Updates that shrink variable length columns
• Large rows that do not pack well: a 5000 bytes row means one row per page, and around 38% of every page is lost mechanically

Note that some free space is normal, and even intentional: the space reserved by the fill factor is internal fragmentation on purpose to absorb future inserts without page splits.

What does it look like?

CREATE DATABASE DemoFrag
GO

ALTER DATABASE DemoFrag SET RECOVERY SIMPLE
GO

USE DemoFrag
GO

CREATE TABLE dbo.SplitDemo (
    Id      int NOT NULL CONSTRAINT PK_SplitDemo PRIMARY KEY CLUSTERED,
    Payload varchar(8000) NOT NULL
);

-- 2 lines about 4000 bytes per page
INSERT INTO dbo.SplitDemo VALUES
(1, REPLICATE('A', 4000)), (2, REPLICATE('B', 4000));

My page is almost full:

Why does it cost ?

Because the same rows are spread over more pages than necessary and everything in SQL Server is done at the page level:

• The buffer pool caches pages, not rows. A page at 40% density wastes 60% of the memory it occupies
• Every read, logical or physical, handles more pages for the same data
• Backups are bigger, integrity checks are longer

This is the key point: internal fragmentation follows the data everywhere, including in RAM. It is a permanent tax on memory, I/O and CPU. However it’s not the case with external fragmentation.

External fragmentation: the offsets do not follow the logical order

At the leaf level of a B-tree, the pages are chained together in the order of the index keys with two pointers stored in every page header (m_nextPage, m_prevPage). This chain is the logical order and it cannot be wrong: it is the order of the keys by definition.

External fragmentation (also called logical fragmentation) means that following this chain no longer corresponds to reading the file sequentially. The pages are logically ordered, but their offsets in the file are not. It takes two forms:

• Holes: the pages of the index are not at consecutive offsets. Pages of other objects, or free pages, sit in between
• Disorder: the pages are at consecutive offsets, but chained in a different order

avg_fragmentation_in_percent counts both forms without distinction. The columns fragment_count and avg_fragment_size_in_pages are closer to the real cost: they measure the length of the contiguous runs.

Where does it come from?

• Several objects growing at the same time: the engine allocates the extents in the order the requests arrive, so the objects interleave in the file. This is the normal life of a database
• Page splits: the new page is allocated where free space is found, rarely next to the original page
• Page deallocations: massive deletes, and the automatic index compaction itself

What does it look like?

USE DemoFrag
GO

CREATE TABLE dbo.SplitDemo (
    Id      int NOT NULL CONSTRAINT PK_SplitDemo PRIMARY KEY CLUSTERED,
    Payload varchar(8000) NOT NULL
);

-- 2 lines about 4000 bytes per page : lines 1 and 2 on 1 page, 3-4 on next page

INSERT INTO dbo.SplitDemo VALUES
(1, REPLICATE('A', 4000)), (2, REPLICATE('B', 4000)),
(3, REPLICATE('C', 4000)), (4, REPLICATE('D', 4000))

-- Lines location (file:page:slot)?
SELECT Id, sys.fn_PhysLocFormatter(%%physloc%%) AS physical_location
FROM dbo.SplitDemo;

My first 2 rows are located on page 376 and the last 2 rows on page 378.

What does page 377 contain? I thought the pages were contiguous?

Since my table has a clustered index on Id it is stored as a B-tree. Can we see it?

SELECT allocated_page_page_id, page_type_desc, page_level, previous_page_page_id, next_page_page_id
FROM sys.dm_db_database_page_allocations(DB_ID(), OBJECT_ID('dbo.SplitDemo'),1, NULL, 'DETAILED');

What does it look like internally?

SELECT * FROM sys.dm_db_page_info (10, 1, 376, DEFAULT);

SELECT * FROM sys.dm_db_page_info (10, 1, 378, DEFAULT);

Page split to generate external fragmentation:

-- Split happens
UPDATE dbo.SplitDemo SET Payload = REPLICATE('B', 4100) WHERE Id = 2;

-- Check page density
SELECT index_level, page_count, avg_page_space_used_in_percent,
avg_fragmentation_in_percent, record_count,
ghost_record_count, version_ghost_record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.SplitDemo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

After the page split:

Line 1 is located at page 376

Line 2 is located at page 379

Line 3 is located at page 378

Line 4 is located at page 378

Why does it cost so little today?

There is exactly one mechanism through which external fragmentation ever costed something: the read-ahead. During a scan, the engine prefetches pages with large I/Os and an I/O is a contiguous range of the file: one offset, one length, up to 512 KB. When the index is contiguous one I/O brings back 64 pages. When it is not the same pages arrive in several smaller I/Os.

When do these smaller I/Os hurt ? Three conditions must be met at the same time:

• The data is cold: a page already in the buffer pool generates no physical I/O at all
• The operation is a large scan: a seek navigates the B-tree by page ID and does not care about the order
• Each additional I/O has a price

On SSD, NVMe and cloud storage, an additional I/O costs a few dozen microseconds and the physical placement below the file is arbitrary anyway (SSD controllers and SANs put the blocks wherever they want).

External fragmentation survived its own usefulness for another reason: it was a good symptom. It increases because of page splits, and page splits destroy the page density and inflate the transaction log.

Reference

Automatic Index Compaction – SQL Server | Microsoft Learn

Thank you. Amine Haloui

L’article SQL Server: Automatic index compaction in Azure (preview) – Part 1 est apparu en premier sur dbi Blog.

UPDATE u SET val = val + 1;

Each new value lands on a value that already exists: 1 wants to become 2, but a 2 is already there; 2 wants to become 3, but a 3 is already there… Every target overlaps a neighboring value. Yet a unique index allows no duplicate, at any instant.
How can this update succeed?

Two tables, one single difference

-- Unique column
CREATE TABLE u (val INT NOT NULL);
CREATE UNIQUE INDEX ux_u ON u(val);
INSERT INTO u (val)
  SELECT TOP (10000) ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
  FROM sys.all_objects a CROSS JOIN sys.all_objects b;

-- Same column, without uniqueness
CREATE TABLE n (val INT NOT NULL);
INSERT INTO n (val) SELECT val FROM u;

The same UPDATE ... SET val = val + 1 on each. Yet the two execution plans differ completely: on n, a direct update, a single modification operator. On u, three unexpected operators appear: Split, Sort, Collapse. That is the entire gap between “any column” and “a unique column”.

Why the naive method doesn’t work

Let’s process u row by row, in ascending order. First row: 1 → 2. But 2 still exists. Two rows carry the value 2: uniqueness is violated immediately. Processing in descending order would get by (10000 → 10001 first, then 9999 → 10000…), but SQL Server can’t bet on the order in which the rows to process are acquired. It needs a method that works regardless of order, and that never lets the same value coexist twice.

Split, sort, collapse

This is the engine’s answer, in three steps. Let’s take five rows with the values 3,4,5,6,7 and follow each step closely.

1. Split (the update becomes delete + insert)

Modifying the key of a unique index directly is risky: the target value may overlap a value that is present. Split works around the problem by decomposing each update into a deletion of the old value followed by an insertion of the new one. Our 5 rows become 10 operations:

DELETE 3     (old value of row #0)
INSERT 4     (new value of row #0)
DELETE 4     (row #1)
INSERT 5
DELETE 5     (row #2)
INSERT 6
DELETE 6     (row #3)
INSERT 7
DELETE 7     (row #4)
INSERT 8

A delete can never fail on uniqueness; an insert fails only if the final value is truly a duplicate. We’ve separated the two risks.

2. Sort (sort to free a value before reoccupying it)

The 10 operations are sorted by the index key. Decisive rule: for equal values, the delete comes before the insert.

DELETE 3
DELETE 4     -- 4 is freed...
INSERT 4     -- ...before being reoccupied
DELETE 5
INSERT 5
DELETE 6
INSERT 6
DELETE 7
INSERT 7
INSERT 8

This is where the overlap is defused. For each value, the old occupant leaves before the new one arrives. At no point do two rows share the same key.

3. Collapse (reduce the intermediate operations)

After sorting, each DELETE k / INSERT k pair for the same value sits adjacent. Collapse fuses each pair into a single in-place update of the existing index entry, rather than deleting and re-inserting it. The two unpaired ends survive as a real delete and a real insert.

DELETE 3          (low end, no partner)
MODIFY 4
MODIFY 5
MODIFY 6
MODIFY 7
INSERT 8          (high end, no partner)

Does the engine really execute it this way? The transaction log settles it.

SELECT Operation, Context, COUNT(*) AS n
FROM sys.fn_dblog(NULL, NULL)
WHERE AllocUnitName LIKE '%ux_u%'
GROUP BY Operation, Context
ORDER BY n DESC;

Result:

Operation            Context               n
LOP_MODIFY_ROW       LCX_INDEX_LEAF        4
LOP_DELETE_ROWS      LCX_MARK_AS_GHOST     1
LOP_SET_BITS         LCX_PFS               1
LOP_INSERT_ROWS      LCX_INDEX_LEAF        1

This result is particularly revealing.

If the five updates had truly been executed as five DELETEs followed by five INSERTs, we would have expected five deletions and five insertions in the log.

Yet what we ultimately observe is:

4x MODIFY_ROW
1x DELETE_ROWS
1x INSERT_ROWS

Collapse is therefore not only a theoretical concept visible in the execution plan. The observations from fn_dblog() show that it translates concretely into a massive reduction of physical operations. The intermediate values are merged into simple index row modifications, while only the two ends of the shift survive as a genuine deletion (3) and a genuine insertion (8).

4. Visualization of the complete SPLIT-SORT-COLLAPSE process

The preceding observations now make it possible to connect the theory to what is actually executed. The following visualization traces the complete sequence of the Split, Sort and Collapse phases, along with the progressive reduction of operations down to the final result observed in the transaction log:

What it costs

The Split / Sort / Collapse mechanism lets SQL Server guarantee uniqueness throughout the entire UPDATE, but this safety comes at a significant cost. To measure this impact, we ran exactly the same update on the two 10,000-row tables created earlier, with STATISTICS IO ON enabled. The results are unambiguous. On table u, SQL Server performed 40,036 logical reads, consumed 266 ms of CPU and 360 ms of elapsed time. On table n, the same operation required only 34 logical reads, 15 ms of CPU and 71 ms of elapsed time.

It’s worth noting that this extra cost corresponds mainly to the logical processing imposed by the uniqueness guarantee. Indeed, our analysis with fn_dblog() shows that the engine does not naively delete and then reinsert each row. After Collapse, the intermediate operations are largely merged into simple MODIFY_ROW operations, which limits the physical work actually performed on the index pages. The price to pay is therefore not so much a full rebuild of the index as a substantial amount of preparatory organization that lets SQL Server guarantee that at no instant do two rows simultaneously hold the same unique key.

A side note about the Halloween Problem

This need to read everything before writing anything has a name. In the mid-1970s, Don Chamberlin, Pat Selinger and Morton Astrahan wrote a query meant to give a 10% raise to every employee earning under $25,000. It ran without error but left everyone at exactly $25,000: the engine scanned the salary index upward, and each raised row crossed the threshold, moved back ahead of the scan, and got raised again. They found it on Halloween, and the name stuck after the day, not the nature of the bug (link).

Their fix was to make sure the optimizer never reads an update through an index built on the very column being updated. Split/Sort/Collapse is a descendant of that idea: when an UPDATE reads and writes the same index, reading and writing must be separated. That’s the Sort’s job. It is a blocking operator: it must consume its entire input before emitting a single row and that property forms the barrier. In doing so it materializes the whole set of rows to modify (in memory via a memory grant or spilling to tempdb if the volume exceeds it). Once that snapshot is frozen, each source row is read exactly once, and no write can ever catch up to it.

Conclusion

Updating a unique key that overlaps itself seems, at first glance, impossible without temporarily violating uniqueness. Yet, thanks to the Split / Sort / Collapse mechanism, SQL Server manages to turn a potentially conflicting operation into a sequence of actions that guarantees no duplicate ever appears at any instant. The execution plan analysis explains the logic the engine uses, while the study of the transaction log shows how this logic materializes concretely. The observations made with fn_dblog() reveal that the intermediate operations are largely reduced to MODIFY_ROW operations. In the end, the real complexity does not lie in the physical modifications themselves, but in the preparatory work needed to preserve uniqueness. That is precisely what explains the overhead observed compared to a classic update: SQL Server pays the price of safety and consistency, but then heavily optimizes the actual execution thanks to Collapse.

This post was inspired by Franck Pachot‘s look at the same problem on Oracle, where he follows the ROWIDs through the index at the physical level. A good companion to this post, to read by the fireplace (or the air-con) !

L’article Split, Sort, Collapse: how SQL Server survives an overlapping unique update est apparu en premier sur dbi Blog.