The Wroclaw Connection

SQLDay 2026 took place this week, from May 11th to 13th, in Wroclaw. Among the featured speakers was Erik Darling, who delivered both a main session and a full-day workshop dedicated to SQL Server performance. During his presentations, he emphasized a concept that is not always widely understood, known as the Row Goal.

The purpose of this article is to recap Erik’s key observations and to introduce this topic, which can serve as a powerful lever for query optimization.

A quick culinary detour and why pierogis matter

In order to understand the explanations below, one key concept must be understood: the Pierogi.

“Pierogi are filled dumplings made from unleavened dough, popular in Polish cuisine and enjoyed worldwide, with various savory and sweet fillings” [1], [2].

To be honest, this has nothing to do with our technical topic, but this dish discovered during this trip is so good that I simply had to include it in this blog.

Filling the aisles and designing our database

In this article, we will use a custom-made database simulating a Polish supermarket selling pierogis. Unfortunately, there aren’t many left, and the product distribution is not uniform. In fact, pierogis account for much less than 1% of the supermarket’s total stock.
Here is the script to create the DB, along with its article reference table and inventory:

USE master;
GO

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'PierogiMart')
    DROP DATABASE PierogiMart;
GO

CREATE DATABASE PierogiMart;
GO

USE PierogiMart;
GO

CREATE TABLE Articles (
    ArticleID INT IDENTITY(1,1) PRIMARY KEY,
    ArticleName VARCHAR(50) NOT NULL,
    Price DECIMAL(10, 2) NOT NULL
);

CREATE TABLE Inventory (
    ReferenceID INT IDENTITY(1,1) PRIMARY KEY,
    ArticleID INT NOT NULL,
    ValidityDate DATETIME NOT NULL,
    Quantity INT NOT NULL,
    CONSTRAINT FK_Article FOREIGN KEY (ArticleID) REFERENCES Articles(ArticleID)
);
GO

INSERT INTO Articles (ArticleName, Price)
VALUES 
('Pierogi', 12.50),
('Pasta', 8.00),
('Sandwich', 6.50),
('Quiche', 9.00);
GO

INSERT INTO Inventory (ArticleID, ValidityDate, Quantity)
SELECT TOP 100000 
    2, 
    DATEADD(DAY, ABS(CHECKSUM(NEWID())) % 365, '2025-01-01'), 
    ABS(CHECKSUM(NEWID())) % 100
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

INSERT INTO Inventory (ArticleID, ValidityDate, Quantity)
SELECT TOP 10000 
    3, 
    DATEADD(DAY, ABS(CHECKSUM(NEWID())) % 365, '2025-01-01'), 
    ABS(CHECKSUM(NEWID())) % 100
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

INSERT INTO Inventory (ArticleID, ValidityDate, Quantity)
SELECT TOP 50000 
    4, 
    DATEADD(DAY, ABS(CHECKSUM(NEWID())) % 365, '2025-01-01'), 
    ABS(CHECKSUM(NEWID())) % 100
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

INSERT INTO Inventory (ArticleID, ValidityDate, Quantity)
SELECT TOP 10 
    1, 
    '2026-12-31', 
    5
FROM sys.all_columns;
GO

We are also including a few indexes to simulate a real-world use case and to support our queries, ensuring we get realistic execution plans:

CREATE NONCLUSTERED INDEX IDX_INV_QUANT ON [dbo].[Inventory] ([Quantity]) include (ArticleID)

CREATE NONCLUSTERED INDEX IDX_INV_VALIDITY on [dbo].[Inventory] ([ValidityDate]) include (ArticleID)

CREATE NONCLUSTERED INDEX IDX_INV_ART on [dbo].[Inventory] (ArticleID)

What exactly is a Row Goal?

Normally, the SQL Server optimizer seeks to minimize the total cost of processing all data for a query. However, if it knows that you only need a specific number of rows (for example, via a TOP, FAST(N), or EXISTS clause), it changes its strategy.

The Row Goal is this specific row target that pushes the optimizer to favor a plan capable of delivering the first few rows as quickly as possible, even if that same plan would be catastrophic for processing the entire table.

TOP(N): Hunting for the best Pierogi

To illustrate the definition above, let’s search for the pierogis with the furthest expiration dates.
Note that the IDX_INV_VALIDITY index supports this query:

SELECT 
    A.ArticleName, 
    A.Price, 
    I.ValidityDate
FROM Articles A
INNER JOIN Inventory I ON A.ArticleID = I.ArticleID
WHERE A.ArticleName = 'Pierogi'
order by I.ValidityDate desc;

SELECT top 10
    A.ArticleName, 
    A.Price, 
    I.ValidityDate
FROM Articles A
INNER JOIN Inventory I ON A.ArticleID = I.ArticleID
WHERE A.ArticleName = 'Pierogi'
order by I.ValidityDate desc

The difference between these two queries is that one requests only the first 10 rows, while the other requests all matching rows. However, this simple distinction is not merely applied when displaying the results; this condition is pushed deeper into the execution plan to influence the choice of operators (Nested Loop, Hash Join, Merge Join) further down the tree.

For the first query, here is the resulting plan:

As we can see, the optimizer chose a Hash Join given the volume of data to be joined. A Hash Match implies that all the data must be read in order to produce the desired result.

For the second query, here is the execution plan:

We can see that this time, the optimizer chose a Nested Loop, which takes each row from the reference table (Inventory) and joins them with the Articles table. This operation can be very time-consuming if a large number of rows must be processed. However, this is where EstimateRowsWithoutRowGoal comes into play. The value of this property is 40’002.5; this means that in a case where a subset of rows was not specifically required, the optimizer would have estimated the number of rows returned by this operator at that value. We can see, however, that the estimation actually used is 10 rows for one execution, a value clearly derived from the TOP(10).

In summary, adding the TOP(10) allowed the optimizer to use a less expensive join for a small amount of data, even though the TOP operator is located at the very end of the execution plan (since a plan is read from right to left).

EXISTS: The search for the first match

As explained previously, the EXISTS clause has a cardinality of 1 because the very first row meeting the internal condition is enough to validate the case. This triggers a Row Goal, as the optimizer must estimate how many rows it will need to read to satisfy (or not) this condition.

Note: In cases where the condition is never met, the optimizer’s plan can become highly inefficient; for full details, see Erik Darling’s blog [here].

We will now observe this behavior with the following query, varying the internal condition of the EXISTS clause by testing one highly selective (discriminant) case and another much less so.

SELECT 
    A.ArticleName, 
    A.Price
FROM Articles A
WHERE not EXISTS (
    SELECT 1/0
    FROM Inventory I 
    WHERE I.ArticleID = A.ArticleID 
    AND I.Quantity > 10 -- vs 98
);

As you may have noticed, I am looking here for products that maintain a certain quantity for every possible consumption date. My goal, of course, is to avoid depleting the stocks of these excellent Polish pierogis so that everyone can enjoy them!

The case where we want to ensure that all existing quantities for an item are greater than 10 is very difficult to satisfy; based on the statistics available to the optimizer, all items have 10 or more units in stock, except for the pierogis!
Since this condition is so widespread, the optimizer knows it will have to scan a large number of rows to find a single case where the condition is not met. This is why it opts for a Scan. This behavior is evidenced by the estimated number of rows to be read (160’010, which represents the entire table).

On the other hand, for a very restrictive condition (quantity > 98), the optimizer recognizes that this condition is highly selective. This is why it favors a Nested Loop, estimating that only 1’608 rows will be necessary to prove the non-existence of the condition.

In summary, EXISTS forces the optimizer to estimate the number of rows required to find a single occurrence that proves whether a condition is met or not, thereby triggering a local optimization of the execution plan.

OPTION(FAST N): Manually steering the engine

The OPTION(FAST N) hint allows you to manually introduce the Row Goal concept into a query. This hint does not limit the total number of results returned; instead, it optimizes the execution plan to retrieve the first N rows as quickly as possible (potentially at the expense of performance for the remaining rows).

In our example below, we have two identical queries retrieving items with a quantity greater than 10. However, the second one uses an execution plan optimized to return the first row as fast as possible (just to make sure no one steals the last available pierogi from the top of the pile!).

select * from Inventory i
where i.Quantity > 10 
order by i.ArticleID

select * from Inventory i
where i.Quantity > 10 
order by i.ArticleID option(fast 1)

Once again, the plans diverge. To retrieve a single row, the IDX_INV_ART index (which already contains sorted ArticleIDs) is used. It performs a Seek on the smallest ArticleID to check if it satisfies the condition of having a quantity greater than 10.

However, by enabling SET STATISTICS TIME ON, we can see that the second execution plan is slower than the first when returning all requested rows (250ms vs. 204ms). While the gap is not massive due to the small table size, the difference is nonetheless observable.

Wrapping up and how to survive the Row Goal gamble

To conclude, the Row Goal is a double-edged sword; brilliant when you only need a quick glimpse of your data, but it can become a real performance trap if the optimizer’s “bet” fails.

Fortunately, if you find that SQL Server is making bad decisions by being too optimistic, you can take back control. By using the hint OPTION (USE HINT ('DISABLE_OPTIMIZER_ROWGOAL')), you force the optimizer to stop daydreaming and focus on the actual cost of the query. It’s the ultimate tool to ensure your execution plan doesn’t end up as messy as a dropped plate of pierogis!

L’article How Row Goal shapes your SQL Server query strategy by hunting for pierogis est apparu en premier sur dbi Blog.

Summer is already around the corner, but it’s not too late for some spring cleaning!
If you manage SQL Server databases with In-Memory tables, you may have already tried to delete a MEMORY_OPTIMIZED_DATA file or FILEGROUP, only to find that SQL Server simply won’t let you.
This limitation has existed since the debut of In-Memory with SQL Server 2014, and the only workaround until now was to recreate the database from scratch.
It is with SQL Server 2025 that Microsoft finally lifts this restriction. In this article, we will analyze these behavioral differences before and after this version.
To conclude, we will draw on the key points presented by Thodoris Katsimanis, DBA Team Technology Manager at Kaizen Gaming, during his session at SQLBits 2026 on In-Memory tables, in order to summarize the challenges and benefits this feature can bring to production.

The Legacy Struggle: In-Memory Limitations from 2014 to 2022

To demonstrate this difference in behavior, we will create a database under SQL Server 2022 with an In-Memory table loaded with data, and then attempt to delete the associated files and FILEGROUP:

DECLARE @DataPath NVARCHAR(512) = '<YOUR_DATA_FOLDER>';
DECLARE @LogPath  NVARCHAR(512) = '<YOUR_LOG_FOLDER>';
DECLARE @SQL      NVARCHAR(MAX);

SET @SQL = N'
CREATE DATABASE TestInMemory
ON PRIMARY (
    NAME = TestInMemory_data,
    FILENAME = ''' + @DataPath + N'TestInMemory.mdf''
),
FILEGROUP XTP_FG CONTAINS MEMORY_OPTIMIZED_DATA (
    NAME = TestInMemory_XTP,
    FILENAME = ''' + @DataPath + N'TestInMemory_XTP''
)
LOG ON (
    NAME = TestInMemory_log,
    FILENAME = ''' + @LogPath + N'TestInMemory_log.ldf''
);';

EXEC sp_executesql @SQL;

USE TestInMemory;
GO

CREATE TABLE dbo.TestTable
(
    ID    INT          NOT NULL,
    Val   NVARCHAR(50) NOT NULL,
    CONSTRAINT PK_TestTable PRIMARY KEY NONCLUSTERED HASH (ID)
        WITH (BUCKET_COUNT = 1024)
)
WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);
GO

INSERT INTO dbo.TestTable VALUES (1, 'Hello'), (2, 'World');
GO

Once our table is loaded (to ensure the table exists and is not just metadata), we can then delete it:

DROP TABLE dbo.TestTable;
GO

SELECT name, type_desc 
FROM sys.tables 
WHERE is_memory_optimized = 1;

The verification clearly shows that no more In-Memory objects exist. We can therefore proceed with the famous cleanup of the files linked to the table we deleted:

ALTER DATABASE TestInMemory 
REMOVE FILE TestInMemory_XTP;
GO

ALTER DATABASE TestInMemory
REMOVE FILEGROUP XTP_FG;
GO

And here is the famous error, impossible to bypass it and sort things out.
Note: This cleanup challenge specifically affects tables using DURABILITY = SCHEMA_AND_DATA, as they are the only ones where data persists within physical files on disk.

SQL Server 2025: Breaking the In-Memory Cleanup Barrier

SQL Server 2025 does not just lift the restriction: it also introduces a new DMV, sys.dm_db_xtp_undeploy_status, which exposes the precise reason why the deletion is not yet possible.
By querying it at the same stage as our previous example, here is what it returns:

USE TestInMemory;
GO

SELECT
    deployment_state,
    deployment_state_desc,
    undeploy_lsn,
    start_of_log_lsn
FROM sys.dm_db_xtp_undeploy_status;
GO

Now we have a clear reason: the start_of_log_lsn is too old, which prevents SQL Server from releasing the FILEGROUP. To resolve this, the LSNs must be advanced. A FULL backup is first required to initialize the backup chain, followed by a LOG backup to effectively advance the position in the logs:

CHECKPOINT;
GO

BACKUP DATABASE TestInMemory TO DISK = 'NUL';
GO

BACKUP LOG TestInMemory TO DISK = 'NUL';
GO

Once the LOG backup is executed and the LSNs are sufficiently advanced, the files can finally be deleted. The sys.dm_db_xtp_undeploy_status view confirms that the XTP engine is no longer deployed and that the cleanup has been successfully performed.

SQL Server 2025 not only introduces the ability to purge empty files that were linked to an In-Memory object but also the ability to troubleshoot their deletion!

From Milliseconds to Microseconds: Thodoris Katsimanis at SQLBits 2026

To conclude this article, let’s look back at the key points covered by Thodoris Katsimanis during his session at SQLBits 2026, entitled “Revolutionizing Database Performance: Deep Dive into SQL InMemory Technology”.
His context is particularly telling: at Kaizen Gaming, SQL Server databases handle thousands of transactions per second in real time, in an environment where every millisecond has a direct impact on the user experience. It is precisely in this type of workload that In-Memory tables reveal their full potential.

Eliminating Page Contention with Latch-Free Architecture

The presentation exposed limitations of the SQL Server engine: in a high-performance system, latches on disk pages (PAGELATCH_EX) create bottlenecks that can lead to Thread Pool exhaustion. The In-Memory architecture solves this problem at its root via a latch-free structure. By relying on optimistic concurrency control and multi-versioning (MVCC), SQL Server no longer waits for locks. Each row has a Begin-Timestamp and an End-Timestamp, allowing transactions to read the valid version of the data without blocking writes.

Maximizing Performance: The Crucial Choice between Hash and BW-Tree

The choice between a Hash index and a Nonclustered index is crucial. The Hash index is perfectly suited for Point Lookups (searches on an exact value): it points directly to the memory address via a hash function. Conversely, the Nonclustered index relies on a BW-Tree structure, which is essential for range scans and sorting, where Hash is of little use. To learn more about indexes for In-Memory tables, check the Microsoft’s documentation.

The Critical Impact of BUCKET_COUNT Misconfiguration

As Thodoris points out, the success of a Hash index relies on tuning the BUCKET_COUNT. This parameter defines the number of entry points in the index. If it is too low, the system generates collision chains: multiple values end up in the same bucket, forcing the engine to scan a linked list, which degrades performance. If it is too high, it consumes memory unnecessarily. Thodoris also highlights a crucial observation: using a Nonclustered index for an equality search can consume significantly more memory than a properly sized Hash index.

Final Thoughts: Embracing the In-Memory Revolution

SQL Server 2025 finally lifts a limitation that has hampered the lifecycle management of In-Memory databases for over ten years. Being able to cleanly delete associated files and FILEGROUPs, understanding why the engine blocks this operation thanks to sys.dm_db_xtp_undeploy_status, and having a clear procedure to remedy it: this is concrete progress for everyone operating this technology in production.

Thodoris Katsimanis’s session at SQLBits 2026 reminds us that the care given to maintenance and monitoring matters just as much as the initial design. In-Memory tables are not a simple performance lever to be activated and forgotten: they require a mastery of their internal mechanisms, thread management to eliminate contention, and rigorous sizing of the BUCKET_COUNT. As he summarizes: the millisecond is no longer a sufficient unit of measurement. In-Memory OLTP aims for the microsecond and in hyper-transactional environments, that is precisely what makes the difference.

L’article SQL Server 2025 In-Memory: New Cleanup Features & SQLBits 2026 Insights est apparu en premier sur dbi Blog.

The Multi-Layered Threat: Why One Tool is Never Enough

We’ve all left the key in our bike lock at least once. This simple human oversight makes the heaviest chain irrelevant and we often see the exact same logic applied to data environments. Most organizations spend months hardening their production core but leave the keys in the locks of the dev and staging systems that sit right next to it.

The numbers back this up. While 91% of organizations are concerned about their exposure across lower environments, a staggering 86% still allow data compliance exceptions in non-production. This gap between concern and action has real consequences: more than half of these organizations have already experienced a breach or audit failure in their testing and development systems (PR Newswire).

Effective security is rarely a single-layer problem. Between the stolen backup that lands in the wrong hands, the analyst running a SELECT on a table they probably shouldn’t see, and the packet quietly crossing an unsecured network segment, the attack surface is wide, and no single mechanism covers it all.

Transport Layer Security (TLS), Transparent Data Encryption (TDE), symmetric encryption, dynamic masking, row-level security, data anonymization: for most RDBMS, the options exist and they work. Most teams already have access to at least one of them. The real challenge isn’t finding a solution; it’s understanding what each one actually protects, where it breaks down, and whether it survives contact with a production environment.

Shadow Environments: The Weakest Link in Your Data Chain

Here is the uncomfortable truth: non-production environments are often where security policies are quietly buried. It starts with a backup restored without encryption, or real customer data seeding a dev database “just for a quick test“.

The fundamental problem is that most protections assume a controlled environment. Encryption can be bypassed by someone with the right credentials. Masking can be misconfigured. Row-level security doesn’t help much when the whole database is sitting on a developer’s laptop.

Technical Trade-offs: Finding Your Strategic Fit

To make this reasoning concrete, the table below maps six core techniques against the operational criteria that define their success. The goal isn’t to pick a favorite tool, but to identify which combination actually addresses your specific vulnerabilities.

	Physical File Theft	Read Access (SELECT)	Network Sniffing	Performance Impact	Granularity	Applicable in Prod (live data)	Applicable in DEV
TLS					Data packet
TDE					Column Tablespace Datafile
Symmetric encryption (applicative)					Field Value
Dynamic Masking					Column
Row-level security					Row
Data anonymization					Field Column

• TLS protects data in motion. The moment a packet leaves a server, TLS ensures anyone intercepting it sees encrypted noise. What it doesn’t do is equally important: it has no opinion about who queries your database or what’s stored on disk. Once the data arrives, TLS’s job is done.
  TLS is now the industry standard for securing data in motion.
  (SQL Server technical blog about TLS here)
• TDE encrypts the physical files that make up your database (data files, log files, backups), so that anyone who gets their hands on them without the encryption key can’t read them. The performance impact is a negligible overhead; in fact, Microsoft for example enables TDE by default for all its cloud-based databases.
  (PostgreSQL technical blog about TDE here)
  However, deploying TDE in development is a security best practice, but it quickly becomes an operational nightmare for environment refreshes, especially if you want to use distinct certificates to avoid leaking production secrets into lower environments.
• Symmetric encryption is field-level encryption applied directly in the application layer. Unlike TDE, it survives a legitimate SELECT; even a user with full read access sees ciphertext unless they hold the applicative key. The tradeoff is performance: encrypting and decrypting at scale adds up quickly.
  (MongoDB technical blog about Client-side Field Level Encryption here)
• Dynamic masking doesn’t encrypt anything. It intercepts query results and replaces sensitive values with masked equivalents based on the user’s role. Fast, lightweight, zero application changes required. The catch: it only controls what’s displayed, not what’s stored. A user with sufficient privileges can bypass it entirely.
  (SQL Server technical blog about dynamic masking here)
• Row-Level Security enforces access at the row level directly inside the database engine. Users see only the rows they’re allowed to see, regardless of how the query is written. No application changes, no trust placed in the calling layer. The policy lives in the database and applies universally.
  (Oracle technical blog about Virtual Private Database here)
• Data anonymization doesn’t protect sensitive data, it eliminates it. Real values are replaced with realistic but fictional equivalents (synthetic data), permanently and irreversibly. No encryption key to steal, no masking rule to bypass. Whatever leaks simply isn’t sensitive anymore. This is why anonymization is the only control that makes unconditional sense in non-production environments. A stolen backup, a misconfigured SELECT, a sniffed packet: none of it matters if the data was anonymized before it ever reached a staging environment. We covered how to implement it in practice in a previous post

Ownership Gaps: The Security No Man’s Land

We are shifting from a technical challenge to a human and organizational one. The security landscape moves so fast that the struggle of mastering every layer has become overwhelming.

This complexity is where governance goes to die. Infrastructure teams build the walls, developers write the code, and DBAs manage the house, but the accountability for the data itself often falls through the cracks. The most dangerous gap isn’t a missing feature; it’s the absence of a governance model strong enough to stop the game of hot potato and force a cross-domain ownership of security.

The CISO’s role in this landscape is not to master every technical layer, it is to force the question of ownership into the open. Who signs off on what data enters a non-production environment? Who is accountable when a dev database is restored without encryption? Who audits that masking policies are still effective after a release?

Without explicit answers to these questions, security becomes a game of assumptions. Every team assumes another layer is holding. And the gaps compound silently, until they don’t.

From Handcrafted Scripts to Enterprise Platforms

Every technique in this table can be implemented on a spectrum, from a carefully written script to a fully automated enterprise solution. The right choice depends on your scale, your team, and how much operational overhead you can realistically absorb.

• TLS certificate deployment: you can generate and rotate certificates manually, instance by instance. Or you can automate the entire lifecycle using Ansible against an internal PKI with a consistent and auditable way that is invisible to the teams consuming it. The security outcome is identical; the operational cost is not.
• Data anonymization: a custom script that detects PII columns and replaces values with masked data works well at small scale. The challenge appears when your data spans multiple database engines (SQL Server, Oracle, PostgreSQL, …) and when anonymized values need to remain consistent across foreign keys and referential constraints. Replacing a customer ID in one table while leaving it intact in another isn’t anonymization, it’s a GDPR incident waiting to happen. Solutions like Delphix Continuous Compliance handle cross-DBMS consistency, constraint awareness, and sensitive field detection out of the box, turning a fragile hand-rolled process into a governed, repeatable and auditable one.
• Dynamic masking and row-level security: defining a handful of policies manually in SSMS is perfectly reasonable for a contained environment. Automating policy deployment across environments and instances is a different challenge entirely. It is a level of scale where ad-hoc scripts quickly become a liability.

Conclusion: Moving Beyond Security by Accident

Security is not a one-time project. It is an operational discipline that requires the same rigor in a developer’s sandbox as it does in production, and that rigor has to be enforced by design, not by goodwill.

Most breaches in non-production environments don’t happen because a tool failed. They happen because nobody owned the decision to use it in the first place.

At dbi services, we help organizations move from fragile, handcrafted scripts to governed, auditable architectures across every environment, every database engine, and every team.

Because under GDPR, one incident is all it takes to make ownership everyone’s problem.

L’article Beyond TDE and TLS: Bridging the Data Security Governance Gap in Lower Environments est apparu en premier sur dbi Blog.

As my colleague Amine Haloui explained in a recent blog post, several strategies exist for migrating an instance. The “Lift and Shift” method (restoring the ReportServer database onto a new instance) is often the most attractive on paper. However, the reality on the ground can be more temperamental.

In some production environments, the target PBIRS instance already exists, hosts its own content, or follows specific configurations that prohibit simply overwriting its underlying ReportServer database. Therefore, we are proceeding here on the premise of a selective and granular migration: we must inject the SSRS catalog into an active PBIRS environment without burning everything to the ground in the process.

When faced with inventories exceeding hundreds or even thousands of reports (RDL), folders, and datasources, a manual approach via the web interface is not an option and automation becomes a necessity.

This article analyzes a systematic approach based on the ReportingServicesTools PowerShell module. The objective is to provide a robust methodology to extract your catalog and redeploy it intelligently, while managing the necessary reconfigurations along the way.

Phase 1: Smart Dumping – Building the Local Staging Area

To migrate cleanly, objects must first be isolated. The idea is not to blindly vacuum everything, but to target the critical folders of your SSRS instance and transform them into flat files (.rdl and .rds) within a local staging area. If your SSRS instance contains specific object types, the scripts can easily be adapted to include them as well.

This is where the power of the SOAP Proxy comes into play. Rather than multiplying slow HTTP calls, we use the native service interface to list and extract our components:

$sourceUrl  = "http://your-ssrs-server/ReportServer"
$exportRoot = "H:\Migration_Dump"

$proxySource = New-RsWebServiceProxy -ReportServerUri $sourceUrl

In a production environment, SSRS folders are often a messy mix of reports, data sources, images, and sometimes obsolete semantic models. To maintain total control over what we export, we isolate the filtering logic.

This Get-AllItemsByType function allows us to retrieve only what truly matters to us, based on the TypeName and file extension returned by the API.

function Get-AllItemsByType {
    param(
        [string]$CurrentPath,
        $Proxy,
        [string]$TypeName 
    )
    try {
        return $Proxy.ListChildren($CurrentPath, $true) | Where-Object { $_.TypeName -eq $TypeName }
    } catch {
        Write-Host "  [!] Error on $CurrentPath : $($_.Exception.Message)" -ForegroundColor Red
        return $null
    }
}

This mapping between the file type and its extension must be defined upfront in a dictionary:

$extensionMap = @{
    "Report" = ".rdl"
    "DataSource" = ".rds"
}

A crucial point in extracting SSRS objects is preserving their context. To ensure a seamless import into PBIRS 2025, we must recreate the exact folder hierarchy of the source server locally.

The trick lies in transforming the SSRS path (formatted as /Folder/SubFolder/Report) into a valid Windows path, while simultaneously handling the extension mapping (.rdl for reports, .rds for DataSources).

function Export-SsrsItems {
    param(
        [string]$RootPath,
        $Proxy,
        [string]$TypeName,
        [string]$ExportRoot
    )

    $items = Get-AllItemsByType -CurrentPath $RootPath -Proxy $Proxy -TypeName $TypeName

    foreach ($item in $items) {
        $relativeItemPath = $item.Path.TrimStart('/').Replace("/", "\")
        $localFilePath    = Join-Path $ExportRoot $relativeItemPath
        $localDirectory   = Split-Path -Path $localFilePath -Parent

        if (-not (Test-Path $localDirectory)) {
            New-Item -ItemType Directory -Path $localDirectory -Force | Out-Null
        }

        Out-RsCatalogItem -Path $item.Path -Destination $localDirectory -Proxy $Proxy
    }
}

By doing this, your H:\Migration_Dump becomes the exact mirror of your SSRS portal. This structural rigor is what will allow us, in the next step, to remap our data sources without having to hunt down which report belongs to which department.

Finally, we define the folders we wish to export along with the document types they contain (since a migration is often the perfect time for a bit of spring cleaning):

$exportTasks = @(
    @{ Path = "/Migration_Source_2"; Types = @("Report") },
    @{ Path = "/Data Sources";  Types = @("DataSource") }
)

Write-Host "--- Selective Export Started ---" -ForegroundColor Cyan

foreach ($task in $exportTasks) {
    foreach ($typeName in $task.Types) {
        $ext = $extensionMap[$typeName]
        Export-SsrsItems `
            -RootPath   $task.Path `
            -Proxy      $proxySource `
            -TypeName   $typeName `
            -Extension  $ext `
            -ExportRoot $exportRoot
    }
}

Phase 2: Data Source Patching – Mass XML Transformation

Once the extraction is complete, you have a local mirror of your source instance, but the data sources still point to the legacy infrastructure.

Instead of manually fixing each connection after the import (the best way to miss half of them), we will apply an automated transformation directly to our local XML files. This allows us to update connection strings in bulk before a single report even hits the target server.

The idea is simple: use PowerShell to inject the new SQL instance wherever necessary, ensuring a functional deployment from the very first second:

$allDataSources = Get-ChildItem -Path $exportRoot -Filter "*.rds" -Recurse

Write-Host "[>] Datasources updated in : $exportRoot" -ForegroundColor Yellow

foreach ($dsFile in $allDataSources) {
    [xml]$xmlContent = Get-Content $dsFile.FullName
 
    $node = $xmlContent.SelectSingleNode("//ConnectString")
    
    if ($null -ne $node) {
        $oldValue = $node."#text" 
        if ($null -eq $oldValue) { $oldValue = $node.InnerText }

        $newValue = $oldValue -replace "OLD_REPORTING_INSTANCE", "NEW_REPORTING_INSTANCE"
        
        if ($oldValue -ne $newValue) {
            $node.InnerText = $newValue
            $xmlContent.Save($dsFile.FullName)
            Write-Host "  [v] ConnectString updated in : $($dsFile.Name)" -ForegroundColor Green
        }
    } else {
        Write-Host "  [!] ConnectString not found in file $($dsFile.Name)" -ForegroundColor Red
    }
}

Moreover, since we are interacting directly with the file’s XML structure, this logic isn’t limited to connection strings: you can apply the same principle to automate changes for any XML property, from timeouts to provider names.

Phase 3: Mass Deployment – Rebuilding the Reporting Portal

At this stage, the operation is purely mechanical. We once again leverage the ReportingServicesTools module to recreate the folder structure and upload the .rds and .rdl files. By following this specific order, PBIRS will automatically restore the links between your reports and their newly patched data sources.

It is worth noting that the script allows for importing into a specific root folder (defined by the $destroot variable). This is particularly useful if you want to isolate the migrated assets into a dedicated directory, such as SSRS_Folder to keep them distinct from the existing hierarchy. Furthermore, this script is designed with safety in mind: it cannot overwrite or delete anything. If a report with the same name already exists in the same location, the -Overwrite:$false argument prevents replacement, ensuring that the import process never destroys existing content.

Here is the final block to complete your migration:

$destUrl   = "http://your-pbirs-server/ReportServer" 
$localDump = "H:\Migration_Dump"
$destRoot  = "/" #Start import in the root folder
$proxyDest = New-RsWebServiceProxy -ReportServerUri $destUrl

$extensionMap = @{
    "Report"     = ".rdl"
    "DataSource" = ".rds"
}

function Ensure-RsFolderBruteForce {
    param($fullFolderPath, $Proxy)
    $parts = $fullFolderPath.Split('/') | Where-Object { $_ -ne '' }
    $currentPath = ''
    
    foreach ($part in $parts) {
        $parent = if ($currentPath -eq '') { "/" } else { $currentPath }
        $target = if ($currentPath -eq '') { "/$part" } else { "$currentPath/$part" }
        
        try {
            $Proxy.CreateFolder($part, $parent, $null) | Out-Null
            Write-Host "  [DIR] Created : $target" -ForegroundColor Cyan
        } catch {
            if ($_.Exception.Message -match "AlreadyExists") {
                # Folder already exists but we continue
            } else {
                Write-Host "  [!] Error for folder $target : $($_.Exception.Message)" -ForegroundColor Red
            }
        }
        $currentPath = $target
    }
}

function Import-SsrsItem {
    param(
        [System.IO.FileInfo]$File,
        [string]$LocalDump,
        [string]$DestRoot,
        $Proxy
    )

    $relativeDir       = $File.DirectoryName.Replace($LocalDump, '').Replace("\", "/")
    $targetFolderPath  = ($DestRoot + $relativeDir).Replace("//", "/")
    $fullItemPath      = ($targetFolderPath + "/" + $File.BaseName).Replace("//", "/")

    Ensure-RsFolderBruteForce -fullFolderPath $targetFolderPath -Proxy $Proxy

    try {
        Write-RsCatalogItem -Path $File.FullName -Destination $targetFolderPath -Proxy $Proxy -Overwrite:$false
        Write-Host "  [DONE] Imported: $fullItemPath" -ForegroundColor Green
    }
    catch {
        if ($_.Exception.Message -match "already exists") {
            Write-Host "  [SKIP] Already created : $fullItemPath" -ForegroundColor Gray
        } else {
            Write-Host "  [FAIL] Error $fullItemPath : $($_.Exception.Message)" -ForegroundColor Red
        }
    }
}

$importOrder = @("DataSource", "Report")

foreach ($typeName in $importOrder) {
    $extension = $extensionMap[$typeName]
    Write-Host "`n[PASS] Import of object with type : $typeName ($extension)" -ForegroundColor Magenta
    
    $filesToImport = Get-ChildItem -Path $localDump -Filter "*$extension" -Recurse

    if ($filesToImport.Count -eq 0) {
        Write-Host "  [i] No file with $extension found." -ForegroundColor Gray
        continue
    }

    foreach ($file in $filesToImport) {
        Import-SsrsItem -File $file -LocalDump $localDump -DestRoot $destRoot -Proxy $proxyDest
    }
}

Write-Host "`nImport done!" -ForegroundColor Green

Importing via SOAP is more resource-intensive than extraction, as the server must validate every piece of metadata and physically recreate the path for each report. On large volumes, this stage can become a bottleneck (averaging ~1 second per report).

To overcome this, we can parallelize the import by folder, creating multiple background jobs running on separate threads. Here is the general skeleton to implement this multi-threaded approach:

$maxJobs = 5 

foreach ($file in $filesToImport) {
    while ((Get-Job -State Running).Count -ge $maxJobs) {
        Start-Sleep -Milliseconds 500
    }

    Start-Job -Name "Import_$($file.Name)" -ScriptBlock {
        param($f, $url, $targetPath)

        $Proxy = New-RsWebServiceProxy -ReportServerUri $url
        
        try {
            Write-RsCatalogItem -Path $f.FullName -Destination $targetPath -Proxy $Proxy -Overwrite:$false
            return "SUCCESS: $($f.Name)"
        } catch {
            return "ERROR: $($f.Name) -> $($_.Exception.Message)"
        }
    } -ArgumentList $file, $destUrl, $targetFolderPath
}

Note : The -Parallel parameter is a feature of the ForEach-Object cmdlet introduced in PowerShell 7 to enable native multi-threading. While it allows for processing multiple objects simultaneously, it is not reliably supported by the ReportingServicesTools library as the underlying API is not thread-safe. To ensure stability and avoid session collisions, it is recommended to use the Start-Job method instead, as it provides better process isolation for each task.

Key Takeaways for a Seamless Cutover

Migrating to Power BI Report Server shouldn’t be a manual challenge. By adopting this PowerShell-driven ETL approach, you replace the uncertainty of manual intervention with industrial-grade rigor.

The primary advantage lies in consistency: regardless of the report volume or folder complexity, the script guarantees an identical and predictable result every single time. By isolating extraction, XML transformation, and ordered importation, you maintain total control over your data integrity.

Ultimately, automation is about securing your delivery and freeing up time for what truly matters: leveraging your data on your brand-new PBIRS 2025 platform.

Happy migrating!

L’article Scaling SSRS Migrations: Multi-Threaded Automation for PBIRS 2025 est apparu en premier sur dbi Blog.

This is the harsh lesson many SQL Server DBAs learned the hard way this week. If your inbox has been suspiciously quiet since your last maintenance window, don’t celebrate just yet: your instance might have simply lost its voice.

CU23: The poisoned gift

The Cumulative Update 23 for SQL Server 2022 (KB5074819), released on January 15 2026, quickly became the “hot” topic on technical forums. The reason? A major regression that purely and simply breaks Database Mail.

Could not load file or assembly 'Microsoft.SqlServer.DatabaseMail.XEvents, Version=17.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.

This error message is the visible part of the iceberg. Beyond the technical crash, it is the silence of your monitoring that should worry you. The real danger here is that while the mailing engine fails, your SQL Agent jobs don’t necessarily report an error. Since the mail items are never even processed, they don’t appear in the failed_items or unsent_items views with typical error statuses. For most monitoring configurations, this means you stay completely unaware that your instance has lost its voice. You aren’t getting alerts, but everything looks fine on the surface.

Database Mail: The (Too) Quiet Hero

We often tend to downplay the importance of Database Mail, treating it as a minor utility. Yet, for many of us, it is the backbone of our monitoring. From SQL Agent job failure notifications to corruption alerts or disk space thresholds, Database Mail is a critical component. When it fails, your infrastructure visibility evaporates, leaving you flying blind in a rather uncomfortable technical fog.

Rollback or Status Quo?

While waiting for an official fix, the best way to protect your production remains a rollback (guidelines available here). Uninstalling a CU is never an easy task: it implies additional downtime and a fair amount of stress. However, staying in total darkness regarding your servers’ health is a much higher risk. Microsoft has promised a hotfix “soon” but in the meantime, a server that reboots is better than a server that suffers in silence.

The Light at the End of the Tunnel (is in the MSDB)

If a rollback is impossible due to SLAs or internal policies, remember that not all is lost. Even if the emails aren’t being sent, the information itself isn’t volatile. SQL Server continues to conscientiously log everything it wants to tell you inside the msdb system tables.

You can query the following table to keep an eye on the alerts piling up:

SELECT mailitem_id, recipients, subject, body, send_request_date, sent_date
 FROM [msdb].[dbo].[sysmail_mailitems]
 where sent_date is null
 and send_request_date > '2026-01-15 00:00:00.000'

It’s less stylish than a push notification, but it’s your final safety net to ensure your Log Shipping hasn’t flatlined or your backups haven’t devoured the last available Gigabyte.

How to Build a Solid “Home-Made” Quick Fix

If you have an SMTP gateway reachable via PowerShell, you can bridge the gap using the native Send-MailMessage cmdlet. This approach effectively bypasses the broken DatabaseMail.exe by using the PowerShell network stack to ship your alerts.

Testing with Papercut

In a Lab environment, you likely don’t have a full-blown Exchange server. To test this script, I recommend using Papercut. It acts as a local SMTP gateway that catches any outgoing mail and displays it in a UI without actually sending it to the internet. Simply run the Papercut executable, and it will listen on localhost:25.

The Recovery Script

For the purpose of this article’s demonstration, the following Central Management Server (CMS) setup is used:

Run the following script as a scheduled task every 15 minutes to replay any messages stuck in the msdb queue:

Import-Module -Name dbatools;

Set-DbatoolsConfig -FullName 'sql.connection.trustcert' -Value $true;

$cmsServer = "SQLAGAD\LAB01"
$cmsGroupName = "LAB_AG"
$smtpServer = "localhost"

try {
    $instances = Get-DbaRegServer -SqlInstance $cmsServer -ErrorAction Stop | 
                 Where-Object group -match $cmsGroupName;

    if (-not $instances) {
        Write-Warning "No instances found in the group '$cmsGroupName' on $cmsServer."
        return
    }

    foreach ($server in $instances) {
        Write-Host "Processing: $($server.Name)" -ForegroundColor Cyan
        
        $query = "SELECT SERVERPROPERTY('ServerName') SQLInstance, * FROM [msdb].[dbo].[sysmail_mailitems] where sent_date is null and send_request_date > DATEADD(minute, -15, GETDATE())"
        
        $blockedMails = Invoke-DbaQuery -SqlInstance $server.Name -Database msdb -Query $query -ErrorAction SilentlyContinue

        if ($blockedMails) {
            foreach ($mail in $blockedMails) {
                Send-MailMessage -SmtpServer $smtpServer `
                                 -From "MSSQLSecurity@lto.com" `
                                 -To $mail.recipients `
                                 -Subject "[CU23-RECOVERY] $($mail.subject)" `
                                 -Body $mail.body `
                                 -Priority High

                Write-Host "Successfully re-sent: $($mail.subject)" -ForegroundColor Green
            }
        }
    }
}
catch {
    Write-Error "Error accessing the CMS: $($_.Exception.Message)"
}

Resulting alerts in the Papercut mailbox:

By leveraging the dbatools module and the native Send-MailMessage function, we can create a “recovery bridge.” The following script is designed to be run as a scheduled task (e.g., every 15 minutes). It scans your entire infrastructure via your CMS, identifies any messages that failed to send in the last 15-minute window, and replays them through PowerShell’s network stack instead of SQL Server’s.

Note on Data Integrity: You will notice that this script intentionally performs no UPDATE or DELETE operations on the msdb tables. We chose to treat the system database as a read-only ‘source of truth’ to avoid any further corruption or inconsistency while the instance is already in an unstable state.

Let’s wrap-up !

Until Microsoft releases an official fix for this capricious CU23, this script acts as a life support system for your monitoring alerts. It is simple, effective, and most importantly, it prevents the Monday morning nightmare of discovering failed weekend backups that went unnoticed because the notification engine was silent.

So, if your SQL alerts prefer staying cozy in the msdb rather than doing their job, you now have the bridge to get them moving. Set up the scheduled task, run the script, and go grab a coffee, your emails are finally back on track.

Until the hotfix lands, keep your scripts ready!

L’article SQL Server 2022 CU23: Database Mail is broken, but your alerts shouldn’t be est apparu en premier sur dbi Blog.

Delphix is a software-defined data platform designed to automate and secure the data lifecycle. To handle data at scale without the usual operational friction, it relies on three pillars:

• Continuous Data (Virtualization) : This virtual appliance ingests production sources to create compressed dSources. You can then provision virtual clones (VDBs) in minutes. It’s not just about optimizing storage via block-sharing, it’s about drastically slashing refresh times, moving from hours of manual DB cloning to a few minutes of automation.
• Continuous Compliance (Masking) : Acting as the compliance safeguard, this virtual appliance anonymizes sensitive data (GDPR, etc.) while keeping referential integrity intact across applications. You get usable test data without the risk of a data leak or a legal headache. For more information, you can read my other blog Data Anonymization as a Service with Delphix Continuous Compliance.
• Data Control Tower (DCT) : The central Control Plane. Unlike the engines above, DCT is a platform that unifies everything through a single API. It also provides a dedicated interface where developers can manually manage their own resources in total autonomy, without needing a DBA behind them for every refresh. It effectively turns isolated instances into a programmable, self-service infrastructure.

The ABAC Approach: Beyond static roles

Access management usually relies on the RBAC (Role-Based Access Control) model. In this setup, permissions are tied to roles (like “Developer” or “Analyst”) and users simply inherit whatever their role allows. While this works fine at the start, it quickly becomes a rigid mess as projects multiply, usually leading to a “role explosion” that’s a nightmare to maintain.

ABAC (Attribute-Based Access Control) takes a different path: rights aren’t frozen into a static role anymore. Instead, they are determined dynamically based on attributes (characteristics of the resource, the user, and the environment). It’s a shift from Who are you? to What are you trying to access and does it match your current context?.

Delphix DCT: Leveraging Tags for Smarter Data Governance

The real strength of Delphix Data Control Tower (DCT) lies in its ability to leverage tags for strict resource segregation. In a sprawling enterprise environment, global visibility is rarely a feature; it’s more of a liability. By mapping specific tags to your resources, you define isolated management boundaries that actually make sense for the business.

This setup ensures that teams only see the infrastructure they are responsible for. It’s a clean way to enforce the Least Privilege principle without the administrative overhead of manual role mapping. By restricting a user’s scope to their specific tags, you eliminate the risk of someone interact with an environment they weren’t even supposed to know existed. It’s about creating a workspace where developers can be autonomous within their own perimeter, while the rest of the infrastructure remains safely out of sight (and out of reach).

ABAC in Action: Configuring Dynamic Access Control in Delphix DCT

Even when managing a massive Delphix environment, you can strictly limit a developer’s or DevOps engineer’s visibility to only the resources they actually own. Through the unified DCT platform, they get a consolidated view of their specific objects only :

In the three images above, each component is identified by the tag Team: BlogTeam. This key-value pair is directly tied to the permissions assigned to the current user. In this specific scenario, a user belonging to the BlogTeam access group is granted the following roles:

While these roles might seem overly permissive at first glance, the ABAC framework in Delphix allows for granular control. It ensures that these permissions are dynamically scoped, applying only to the specific resources that match the user’s attributes.

It is important to understand that ABAC in Delphix DCT isn’t just a “view-only” filter ; it doesn’t just dictate who sees what, it defines who can do what.

By combining roles with tag-based scopes, you can achieve surgical precision in your permissions. For instance, a developer might be granted the Provision VDBs right on all resources tagged with Team : BlogTeam, while only having Refresh or Snapshot capabilities on VDBs tagged with Environment: Integration. This ensures that even within the same project, critical actions are restricted to the right people at the right time, without ever needing to touch a single manual access list.

The true power of Delphix DCT lies in its ability to transform a basic administrative task (tagging) into a robust automated governance engine. It provides administrators with absolute control while ensuring developers operate within a streamlined interface, strictly confined to their specific workspace.

DCT: Moving from GUI Management to API-First Governance

Integrating Delphix into a modern CI/CD pipeline means saying a final goodbye to manual clicks. Thanks to DCT’s API-First architecture, every resource is treated as a programmable object. By using tags as dynamic filters, you stop managing environments by their names or IP addresses and start orchestrating them by their business properties.

The real beauty here is the ability to script complex workflows in just a few lines of code. Instead of manually triggering a refresh for ten different test databases (a task as tedious as it is prone to error) your pipeline can simply query the API to identify every resource tagged with Team : BlogTeam and fire off the operation simultaneously. This approach doesn’t just save precious time; it ensures total reproducibility while eliminating the human factor.

To give you a concrete example, the following Python snippet shows how to authenticate with the DCT API and retrieve details for all VDBs associated with the specific tag Team : BlogTeam :

import requests, urllib3, logging
from typing import List, Dict, Any

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger(__name__)

class Config:
    dct_url = "https://<DELPHIX_ENGINE_URL>/dct/v3"
    username = "YOUR_USERNAME"
    password = "YOUR_PASSWORD"

class DctApiClient:
    def __init__(self, cfg):
        self.cfg = cfg
        self.token = None

    def authenticate(self):
        try:
            r = requests.post(f"{self.cfg.dct_url}/login", 
                json={"username": self.cfg.username, "password": self.cfg.password}, verify=False)
            self.token = r.json()["access_token"]
            logger.info(f"Connected (expires in {r.json().get('expires_in')}s)")
            return True
        except Exception as e:
            logger.error(f"Authentication failed: {e}")
            return False

    def headers(self):
        return {"Authorization": f"Bearer {self.token}", "Accept": "application/json"}

    def get_vdbs_by_tag(self, tag_key: str, tag_value: str) -> List[Dict[str, Any]]:
        vdbs = requests.get(f"{self.cfg.dct_url.replace('/dct/v3', '/v3')}/vdbs", 
                           headers=self.headers(), verify=False).json().get('items', [])
        
        filtered = [v for v in vdbs if any(t.get('key')==tag_key and t.get('value')==tag_value 
                                           for t in v.get('tags', []))]
        logger.info(f"{len(filtered)} VDB(s) found")
        
        for v in filtered:
            status_color = '\033[92m' if v.get('status','').lower()=="running" else '\033[91m'
            print(f"\n{'='*35}\n{v.get('name'):20} | {status_color}{v.get('status')}\033[0m\n"
                  f"ID: {v.get('id')}\nType: {v.get('database_type')}\n{'='*35}")
        return filtered

if __name__ == "__main__":
    client = DctApiClient(Config)
    if client.authenticate():
        client.get_vdbs_by_tag("Team", "BlogTeam")

And the result :

INFO: Connected (expires in 86400s)
INFO: 1 VDB(s) found

===================================
VDB_BLOG             | RUNNING
ID: 1-MSSQL_DB_CONTAINER-190
Type: MSSql
===================================

While Python offers great flexibility for complex logic, the same result can be achieved more concisely using the dct-toolkit. This native CLI tool simplifies interactions with the DCT engine, allowing you to execute the same resource filtering with a single command:

dct-toolkit search_vdbs -c name,tags filter_expression "tags CONTAINS {key EQ 'team' AND value EQ 'BlogTeam'} "

Conclusion: Bridging Governance and Agility

In summary, implementing ABAC within Delphix Data Control Tower marks the end of an era where security was synonymous with administrative friction. By using tags as the foundation of your governance, you transform a complex data infrastructure into a granular, secure self-service ecosystem. It is no longer the tool dictating your processes, but your business requirements dynamically driving access control.

A major advantage of this model is the seamless onboarding of new developers. Instead of manually configuring complex permissions for every newcomer, simply assigning the appropriate tags allows them to be productive instantly, with immediate and secure visibility over their specific scope.

This approach opens the door to advanced use cases that we have only scratched the surface of here:

• Automated Self-Service: Integrating tag creation directly into your provisioning processes so that resources are immediately isolated within the correct perimeter.
• FinOps & Showback: Leveraging this same tagging logic to accurately allocate storage and compute costs by project or team.
• Dynamic Data Masking: Automating masking jobs based on tag criticality (e.g., Criticality : Confidential), ensuring no sensitive data leaves the production environment without protection.

And you? How are you managing access in your non-production environments for developers? Is API automation already at the heart of your DataOps strategy? Let’s discuss it in the comments, or feel free to reach out to explore these topics further.

L’article Scaling Data Governance: Why ABAC is My Favorite Delphix DCT Feature est apparu en premier sur dbi Blog.

Monitoring backups in a SQL Server Always On Availability Group can often feel like a game of hide and seek where the rules change every time you flip a failover switch. On paper, your backup strategy is solid ; you’ve configured your replica priorities and your jobs are running like clockwork. But when you query the backup history to ensure everything is under control, you realize the truth is fragmented.

Because the msdb database is local to each instance, each replica only knows about the backups it performed itself. If your backups happen on the secondary today and the primary tomorrow, no single node holds the complete story. This leads to inconsistent monitoring reports, “false positive” alerts, and a lot of manual jumping between instances just to answer a simple question: “Are we actually protected with consistent backups?”.

Before we dive deeper, a quick disclaimer for the lucky ones: If you are running SQL Server 2022 and have implemented Contained Availability Groups you can stop reading here, you are safe. In a Contained AG, the system databases are replicated alongside your data, meaning the backup history is finally unified and follows the group.

For the others, in this post, we’re going to explore how to stop chasing metadata across your cluster. We’ll look at the limitations of the local msdb, compare different ways to consolidate a unified backup view using Linked Servers, OPENDATASOURCE, and PowerShell, and see how to build a monitoring query that finally tells the whole truth, regardless of where the backup actually ran.

The traditional approach

The traditional approach is therefore to query the msdb through the AG listener, pointing to the primary replica. Using a simple TSQL query to find the most recent backup, we get the following results.

SELECT
bs.database_name,
MAX(bs.backup_finish_date) AS LastBackup
FROM msdb.dbo.backupset bs
WHERE bs.database_name = 'StackOverflow2010'
GROUP BY bs.database_name;

As we can see, the two backup dates differ (14:39 being the time the database was created in the AG and 15:15 being the time of the last backup). As a result, if a failover occurs, the main msdb will contain incomplete data because the primary msdb will be the one on SQLAGVM2.

This traditional approach is perfectly acceptable for standalone instances because there is only one source of truth, but in the case of Always-On, we need a more robust solution. Let’s examine the different possibilities.

The good old Linked Servers

If you want to keep everything within the SQL Engine, Linked Servers are your go-to tool. The idea is simple: from your Primary replica, you reach out to the Secondary, query its msdb, and union the results with your local data.

SELECT
@@SERVERNAME AS [PrimaryServer],
bs.database_name,
MAX(bs.backup_finish_date) AS LastBackup
FROM msdb.dbo.backupset bs
WHERE bs.database_name = 'StackOverflow2010'
GROUP BY bs.database_name

UNION ALL

SELECT
'REMOTE_MSDB' AS [ReportingServer],
bs.database_name,
MAX(bs.backup_finish_date) AS LastBackup
FROM [REMOTE_MSDB].msdb.dbo.backupset bs
WHERE bs.database_name = 'StackOverflow2010'
GROUP BY bs.database_name;

On paper, the best way to secure a Linked Server is to use the Self option, ensuring that your own permissions are carried over to the next node. It’s the most transparent approach for auditing and security. However, this is where we often hit a silent wall: the Double-Hop. Unless Kerberos delegation is perfectly configured in your domain, NTLM will prevent your identity from traveling to the second replica. You’ll end up with a connection error, not because of a lack of permissions, but because your identity simply couldn’t make the trip. To determine the type of authentication protocol you are using, use the following query.

SELECT auth_scheme 
FROM sys.dm_exec_connections 
WHERE session_id = @@SPID;

To bypass this hurdle, it is common to see Fixed Logins being used as a pragmatic workaround. But by hardcoding credentials to make the bridge work, we create a permanent, pre-authenticated tunnel. From a security standpoint, this can facilitate lateral movement if one instance is ever compromised, a major concern in modern Zero Trust architectures. Furthermore, it obscures your audit logs, as the remote server only sees the service account instead of the actual user. These hidden complexities and security risks are precisely why many DBAs are now moving toward more decoupled, scalable alternatives.

The “On-the-Fly” Connection: OPENDATASOURCE

To address the frustrations of Linked Servers, another T-SQL path often explored is the use of ad hoc queries via OPENDATASOURCE.
The concept is tempting: instead of building a permanent bridge (Linked Server), you use a temporary ladder ; OPENDATASOURCE allows you to define a connection string directly inside your T-SQL statement, reaching out to a remote replica only for the duration of that specific query. It feels lightweight and dynamic because it requires no pre-configured server objects.

In theory, you would use it like this to pull backup history from your secondary node:

SELECT 
    @@SERVERNAME AS [Source Server],
    bs.database_name, 
    MAX(bs.backup_finish_date) AS LastBackup
FROM OPENDATASOURCE('MSOLEDBSQL', 'Data Source=SQLAGVM1,1432;Integrated Security=SSPI').msdb.dbo.backupset bs
WHERE bs.database_name = 'StackOverflow2010'
GROUP BY bs.database_name;

However, if the environment is not properly configured, an error will occur immediately.

Msg 15281, Level 16, State 1, Line 1
SQL Server blocked access to STATEMENT 'OpenRowset/OpenDatasource' of component 'Ad Hoc Distributed Queries' because this component is turned off as part of the security configuration for this server. A system administrator can enable the use of 'Ad Hoc Distributed Queries' by using sp_configure. 

By default, SQL Server locks this door. It is a protective measure to prevent users from using the SQL instance as a jumping point to query any other server on the network. To get past this error, a sysadmin must explicitly enable Ad Hoc Distributed Queries. This requires tweaking the advanced configuration of the instance.

EXEC sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
EXEC sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO

Once these options have been reconfigured, access to the remote msdb is finally possible.

While we successfully made OPENDATASOURCE work, the trade-offs are significant. We have ended up with a solution that is brittle, difficult to maintain, and a potential liability:

• Hardcoding & Maintenance: Every replica requires a manually adapted connection string. If a server is renamed, a port is migrated, or a password expires, the entire monitoring logic collapses.
• Security & Shadow IT Risk: Enabling Ad Hoc Distributed Queries opens a permanent hole in your instance. You aren’t just allowing your script to run; you are allowing any sysadmin to connect to any external server, creating a ghost feature that can be easily misused.

In short, we are fighting the SQL engine’s security defaults just to get a simple timestamp. For a truly robust and scalable solution, it is time to look beyond T-SQL.

Here comes the superhero Powershell

PowerShell steps in as the ultimate lifesaver, delivering high-level automation and pure execution simplicity. It allows you to centralize and control your scripts from an external management server, piloting your entire fleet remotely without cluttering your SQL instances.

By leveraging the power of dbatools, we shatter the limitations of traditional T-SQL. We gain dynamic flexibility and enhanced security by bypassing risky configurations, all while maintaining total control over how we manipulate the retrieved data. The security gain does not come from PowerShell itself, but from removing risky SQL Server surface area features and centralizing access control on a hardened management host. This works because PowerShell establishes direct connections from your management host to each replica. This bypasses the NTLM double-hop issue entirely, as your identity is never passed between servers, removing any need for complex Kerberos delegation or risky fixed logins.

Here is how to put this into practice. By using the Availability Group Listener as your unique gateway, you can dynamically discover the cluster topology and query all member nodes.

$Listener = 'SQLLISTENER,1432'
$TargetDB = 'StackOverflow2010'

try {
    $Nodes = Invoke-DbaQuery -SqlInstance $Listener -Database 'master' -Query "SELECT replica_server_name FROM sys.dm_hadr_availability_replica_cluster_states" 
    $BackupQuery = "SELECT SERVERPROPERTY('ServerName') as [InstanceName], MAX(backup_finish_date) as [LastBackup] FROM msdb.dbo.backupset WHERE database_name = '$TargetDB' GROUP BY database_name"
    write-output $Nodes
    $Results = foreach ($Node in $Nodes.replica_server_name) {
        write-output $Node
        Invoke-DbaQuery -SqlInstance $Node -Database 'msdb' -Query $BackupQuery -ErrorAction SilentlyContinue
    }

    $Results | Where-Object { $_.LastBackup } | Sort-Object LastBackup -Descending | Select-Object -First 1 | Format-Table -AutoSize
}
catch {
    Write-Error "Error : $($_.Exception.Message)"
}

This script is just a first step, but it lays the foundation for truly scalable infrastructure management. By shifting the logic to PowerShell instead of overloading our SQL instances, we achieve a robust and extensible method capable of handling large Always-On ecosystems without additional manual effort.

In this example, the listener name is hardcoded for the sake of clarity. However, the true strength of this approach lies in its ability to work behind the scenes with a dynamic inventory. In a Production environment, you would typically query a CMDB or a centralized configuration file to automatically populate the list of instances. This transforms a simple check into a silent, reliable automation that adapts seamlessly as your SQL environment evolves.

While we wrote the T-SQL manually in this example for the sake of clarity, it is worth noting that dbatools offers an even more streamlined approach with the Get-DbaBackupHistory command. This native function eliminates the need for manual queries entirely, returning rich metadata objects that are ready to be filtered and aggregated across your entire fleet.

$Nodes.replica_server_name | Get-DbaBackupHistory -Database 'StackOverflow2010' | Sort-Object End -Descending | Select-Object -First 1

Final Thoughts: Taking Control of the Always-On Fleet

To wrap up, remember that technical skills are only as good as the management strategy behind them. Transitioning away from legacy methods toward a PowerShell-driven approach is about gaining control over your environment. Here is what you should keep in mind:

• Beyond T-SQL Boundaries: While Linked Servers or OPENDATASOURCE might work for quick fixes, they quickly become bottlenecks and security risks in hardened infrastructures.
• Object-Oriented Efficiency: By using PowerShell and dbatools, you stop managing raw text and start handling objects. This allows you to effortlessly filter, sort, and aggregate data from multiple Always-On nodes to extract a single, reliable source of truth.
• Smarter Security: Running queries externally via dedicated management shells ensures you maintain a high security posture without needing to enable high-risk surface area features on your SQL instances.

The real game-changer is the Central Management Server. By centralizing your logic on a dedicated administration machine, you stop scattering scripts across every instance. This server becomes your orchestrator: it pulls from your inventory (CMDB, central tables), broadcasts tasks across your entire fleet, and consolidates the results. This is exactly the approach we take at dbi services to manage the extensive SQL Server environments under our care. We leverage PowerShell and the dbatools module as the backbone of our scripting architecture. This allows us to collect data in the most comprehensive and optimized way possible, ensuring we deliver top-tier service to our clients.

This is how you move from artisanal, server-by-server management to an industrial-grade automation capable of piloting hundreds of instances with the same simplicity as a single one.

L’article SQL Server Always-On: Centralizing Backup History Across Replicas est apparu en premier sur dbi Blog.

Why and How to Anonymize Data

Development teams require realistic datasets in order to:

• Test application performance
• Validate complex business processes
• Reproduce error scenarios
• Train Business Intelligence or Machine Learning algorithms

However, the use of real data requires the implementation of anonymization or pseudonymization mechanisms ensuring:

• Preservation of functional and referential consistency
• Prevention of data subject re-identification

Among the possible anonymization techniques, the main ones include:

• Dynamic Data Masking, applied on-the-fly at access time but which does not anonymize data physically
• Tokenization, which replaces a value with a surrogate identifier
• Cryptographic hashing, with or without salting

Direct data copy from Production to Development

In this scenario, a full backup of the production database is restored into a development environment. Anonymization is then applied using manually developed SQL scripts or ETL processes.

This approach presents several critical weaknesses:

• Temporary exposure of personal data in clear text
• Lack of formal traceability of anonymization processes
• Risk of human error in scripts
• Non-compliance with GDPR requirements

This model should therefore be avoided in regulated environments.

Data copy via a Staging Database in Production

This model introduces an intermediate staging database located within a security perimeter equivalent to that of production. Anonymization is performed within this secure zone before replication to non-production environments.

This approach makes it possible to:

• Ensure that no sensitive data in clear text leaves the secure perimeter
• Centralize anonymization rules
• Improve overall data governance

However, several challenges remain:

• Versioning and auditability of transformation rules
• Governance of responsibilities between teams (DBAs, security, business units)
• Maintaining inter-table referential integrity
• Performance management during large-scale anonymization

Integration of Delphix Continuous Compliance

In this architecture, Delphix is integrated as the central engine for data virtualization and anonymization. The Continuous Compliance module enables process industrialization through:

• An automated data profiler identifying sensitive fields
• Deterministic or non-deterministic anonymization algorithms
• Massively parallelized execution
• Orchestration via REST APIs integrable into CI/CD pipelines
• Full traceability of processing for audit purposes

This approach enables the rapid provisioning of compliant, reproducible, and secure databases for all technical teams.

Conclusion

Database anonymization should no longer be viewed as a one-time constraint but as a structuring process within the data lifecycle. It is based on three fundamental pillars:

• Governance
• Pipeline industrialization
• Regulatory compliance

An in-house implementation is possible, but it requires a high level of organizational maturity, strong skills in anonymization algorithms, data engineering, and security, as well as a strict audit framework. Solutions such as Delphix provide an industrialized response to these challenges while reducing both operational and regulatory risks.

To take this further, Microsoft’s article explaining the integration of Delphix into Azure pipelines analyzes the same issues discussed above, but this time in the context of the cloud : Use Delphix for Data Masking in Azure Data Factory and Azure Synapse Analytics

What’s next ?

This use case is just one example of how Delphix can be leveraged to optimize data management and compliance in complex environments. In upcoming articles, we will explore other recurring challenges, highlighting both possible in-house approaches and industrialized solutions with Delphix, to provide a broader technical perspective on data virtualization, security, and performance optimization.

What about you ?

How confident are you about the management of your confidential data?
If you have any doubts, please don’t hesitate to reach out to me to discuss them !

L’article Data Anonymization as a Service with Delphix Continuous Compliance est apparu en premier sur dbi Blog.

In this post, we’re going to look at what really happens when you try to use nested transactions in SQL Server. We’ll walk through a dead-simple demo, expose why @@TRANCOUNT is more illusion than isolation, and see how a single rollback can quietly unravel your entire call chain. If you’ve ever assumed nested transactions can behave the same way as in Oracle for example, this might clarify a few things you didn’t expect !

Practical example

Before diving into the demonstration, let’s set up a simple table in tempdb and illustrate how nested transactions behave in SQL Server.

IF OBJECT_ID('tempdb..##DemoLocks') IS NOT NULL
    DROP TABLE ##DemoLocks;

CREATE TABLE ##DemoLocks (id INT IDENTITY, text VARCHAR(50));

BEGIN TRAN MainTran;

BEGIN TRAN InnerTran;
INSERT INTO ##DemoLocks (text) VALUES ('I''m just a speedy insert ! Nothing to worry about');
COMMIT TRAN InnerTran;

WAITFOR DELAY '00:00:10';

ROLLBACK TRAN MainTran;

Let’s see how locks behave after committing the nested transaction and entering the WAITFOR phase. If nested transactions provided isolation between each other, no locks should remain since the transaction no longer works on any object. The following query shows all locks associated with my query specifically and the ##Demolocks table we are working on.

SELECT 
    l.request_session_id AS SPID,
    r.blocking_session_id AS BlockingSPID,
    resource_associated_entity_id,
    DB_NAME(l.resource_database_id) AS DatabaseName,
    OBJECT_NAME(p.object_id) AS ObjectName,
    l.resource_type AS ResourceType,
    l.resource_description AS ResourceDescription,
    l.request_mode AS LockMode,
    l.request_status AS LockStatus,
    t.text AS SQLText
FROM sys.dm_tran_locks l
LEFT JOIN sys.dm_exec_requests r
    ON l.request_session_id = r.session_id
LEFT JOIN sys.partitions p
    ON l.resource_associated_entity_id = p.hobt_id
OUTER APPLY sys.dm_exec_sql_text(r.sql_handle) t
where t.text like 'IF OBJECT%'
    and OBJECT_NAME(p.object_id) = '##DemoLocks'
ORDER BY l.request_session_id, l.resource_type;

And the result :

All of this was just smoke and mirrors !
We clearly see in the image 2 persistent locks of different types:

• LockMode IX: Intent lock on a data page of the ##DemoLocks table. This indicates that a lock is active on one of its sub-elements to optimize the engine’s lock checks.
• LockMode X: Exclusive lock on a RID (Row Identifier) for data writing (here, our INSERT).
  
  For more on locks and their usage : sys.dm_tran_locks (Transact-SQL) – SQL Server | Microsoft Learn

In conclusion, SQL Server does not allow nested transactions to maintain isolation from each other, and causes nested transactions to remain dependent on their main transaction, which prevents the release of locks. Therefore, the rollback of MainTran causes the above query to leave the table empty, even with a COMMIT at the nested transaction level. This behavior still respects the ACID properties (Atomicity, Consistency, Isolation, and Durability), which are crucial for maintaining data validity and reliability in database management systems.

Now that we have shown that nested transactions have no useful effect on lock management and isolation, let’s see if they have even worse consequences. To do this, let’s create the following code and observe how SQL Server behaves under intensive nested transaction creation. This time, we will add SQL Server’s native @@TRANCOUNT variable, which allows us to analyze the number of open transactions currently in progress.

 CREATE PROCEDURE dbo.NestedProc
    @level INT
AS
BEGIN
    BEGIN TRANSACTION;

    PRINT 'Level ' + CAST(@level AS VARCHAR(3)) + ', @@TRANCOUNT = ' + CAST(@@TRANCOUNT AS VARCHAR(3));

    IF @level < 100
    BEGIN
        SET @level += 1
        EXEC dbo.NestedProc @level;
    END

    COMMIT TRANSACTION;
END
GO

EXEC dbo.NestedProc 1;

This procedure recursively creates 100 nested transactions, if we manage to go that far… Let’s look at the output.

Level 1, @@TRANCOUNT = 1
[...]
Level 32, @@TRANCOUNT = 32

Msg 217, Level 16, State 1, Procedure dbo.NestedProc, Line 12 [Batch Start Line 15]
Maximum stored procedure, function, trigger, or view nesting level exceeded (limit 32).

Indeed, SQL Server imposes various limitations on nested transactions which imply that if they are mismanaged, the application may suddenly suffer a killed query, which can be very dangerous. These limitations are in place to act as safeguards against infinite nesting loops of nested transactions.
Furthermore, we see that @@TRANCOUNT increments with each new BEGIN TRANSACTION, but it does not reflect the true number of active main transactions; i.e., there are 32 transactions ongoing but only 1 can actually release locks.

Ok, but we still didn’t see what a real nested transaction would look like !

I understand, we cannot stop here. I need to go get my old Oracle VM from my garage and fire it up.
Oracle has a pragma called AUTONOMOUS_TRANSACTION that allows creating independent transactions inside a main transaction. Let’s see this in action with a small code snippet.

CREATE TABLE test_autonomous (
    id NUMBER PRIMARY KEY,
    msg VARCHAR2(100)
);
/

CREATE OR REPLACE PROCEDURE auton_proc IS
    PRAGMA AUTONOMOUS_TRANSACTION;
BEGIN
    INSERT INTO test_autonomous (id, msg) VALUES (2, 'Autonomous transaction');
    COMMIT;
END;
/

CREATE OR REPLACE PROCEDURE main_proc IS
BEGIN
    INSERT INTO test_autonomous (id, msg) VALUES (1, 'Main transaction');
    auton_proc;
    ROLLBACK;
END;
/

In this code, we create two procedures:

• main_proc, the main procedure, inserts the first row into the table.

• auton_proc, called by main_proc, adds a second row to the table.

auton_proc is committed while main_proc is rolled back. Let’s observe the result:

SQL> SELECT * FROM test_auton;

        ID MSG
---------- --------------------------------------------------
         2 Autonomous transaction

Now, that is true isolation between transactions! Here, the inner transaction achieves transactional independence and can persist regardless of the fate of its main transaction. While autonomous transactions in Oracle are not qualified as purely nested because they do not share a common locking context, they serve the same architectural purpose as the one we are demonstrating in this article: allowing a sub-unit of work to decouple its success from the parent flow.

Summary

In summary, SQL Server and Oracle can handle transactions in different ways. In SQL Server, nested transactions do not create real isolation: @@TRANCOUNT may increase, but a single main transaction actually controls locks and the persistence of changes. Internal limits, like the maximum nesting of 32 procedures, show that excessive nested transactions can cause critical errors.

In contrast, Oracle, through PRAGMA AUTONOMOUS_TRANSACTION, allows for truly independent transactions launched from within a main flow. These autonomous transactions can be committed or rolled back without affecting the parent transaction, providing a practical mechanism to decouple specific tasks from the main transactional outcome.

As Brent Ozar points out, SQL Server also has a SAVE TRANSACTION command, which allows you to save a state after a nested transaction has been committed, for example. This command therefore provides more flexibility in managing nested transactions but does not provide complete isolation of sub-transactions. Furthermore, as Brent Ozar emphasizes, this command is complex and requires careful analysis of its behavior and the consequences it entails.
Another approach to bypass SQL Server’s nested-transaction limitations is to manage transaction coordination directly at the application level, where each logical unit of work can be handled independently.

The lesson is clear: appearances can be deceiving! Understanding the actual behavior of transactions in each DBMS is crucial for designing reliable applications and avoiding unpleasant surprises.

L’article The truth about nested transactions in SQL Server est apparu en premier sur dbi Blog.

Working with XML in SQL Server can feel like taming a wild beast. It’s undeniably flexible and great for storing complex hierarchical data, but when it comes to querying efficiently, many developers hit a wall. That’s where things get interesting.

In this post, we’ll dive into a real-world scenario with half a million rows, put two XML query methods head-to-head .exist() vs .value(), and uncover how SQL Server handles them under the scenes.

Practical example

To demonstrate this, we’ll use SQL Server 2022 Developer Edition and create a table based on the open-source StackOverflow2010 database, derived from the Posts table, but storing part of the original data in XML format. We will also add a few indexes to simulate an environment with a minimum level of optimization.

CREATE TABLE dbo.PostsXmlPerf
(
    PostId        INT           NOT NULL PRIMARY KEY,
    PostTypeId    INT           NOT NULL,
    CreationDate  DATETIME      NOT NULL,
    Score         INT           NOT NULL,
    Body          NVARCHAR(MAX) NOT NULL,
    MetadataXml   XML           NOT NULL
);

INSERT INTO dbo.PostsXmlPerf (PostId, PostTypeId, CreationDate, Score, Body, MetadataXml)
SELECT TOP (500000)
       p.Id,
       p.PostTypeId,
       p.CreationDate,
       p.Score,
       p.Body,
       (
           SELECT  
               p.OwnerUserId     AS [@OwnerUserId],
               p.LastEditorUserId AS [@LastEditorUserId],
               p.AnswerCount     AS [@AnswerCount],
               p.CommentCount    AS [@CommentCount],
               p.FavoriteCount   AS [@FavoriteCount],
               p.ViewCount       AS [@ViewCount],
               (
                   SELECT TOP (5)
                          c.Id           AS [Comment/@Id],
                          c.Score        AS [Comment/@Score],
                          c.CreationDate AS [Comment/@CreationDate]
                   FROM dbo.Comments c
                   WHERE c.PostId = p.Id
                   FOR XML PATH(''), TYPE
               )
           FOR XML PATH('PostMeta'), TYPE
       )
FROM dbo.Posts p
ORDER BY p.Id;

CREATE nonclustered INDEX IX_PostsXmlPerf_CreationDate
ON dbo.PostsXmlPerf (CreationDate);

CREATE nonclustered INDEX IX_PostsXmlPerf_PostTypeId
ON dbo.PostsXmlPerf (PostTypeId);

Next, let’s create two queries designed to interrogate the column that contains XML data, in order to extract information based on a condition applied to a value stored within that XML.

SET STATISTICS IO ON;
SET STATISTICS TIME ON;

DBCC FREEPROCCACHE;

SELECT PostId, Score
FROM dbo.PostsXmlPerf
WHERE MetadataXml.exist('/PostMeta[@OwnerUserId="8"]') = 1;

DBCC FREEPROCCACHE;

SELECT PostId, Score
FROM dbo.PostsXmlPerf
WHERE MetadataXml.value('(/PostMeta/@OwnerUserId)[1]', 'INT') = 8

Comparing logical and physical reads, we notice something interesting:

At first glance, the number of pages read is identical, but .exist() is clearly taking more time. Why? Execution plans reveal that .exist() sneaks in a Merge Join, adding overhead.
Additionally, on both execution plans we can see a small yellow bang icon. On the first plan, it’s just a memory grant warning, but the second one is more interesting:

Alright, a bit strange ; let’s move forward with some tuning and maybe this warning will disappear.
To help with querying, it can be useful to create a more targeted index for XML queries.
Let’s create an index on the column that contains XML. However, as you might expect, it’s not as straightforward as indexing a regular column. For an XML column, you first need to create a primary XML index, which physically indexes the overall structure of the column (similar to a clustered index), and then a secondary XML index, which builds on the primary index and is optimized for a specific type of query (value, path, or property) – to know more about XML indexes : Microsoft Learn, MssqlTips.
So, let’s create these indexes !

CREATE PRIMARY XML INDEX IX_XML_Primary_MetadataXml
ON dbo.PostsXmlPerf (MetadataXml);

CREATE XML INDEX IX_XML_Value_MetadataXml
ON dbo.PostsXmlPerf (MetadataXml)
USING XML INDEX IX_XML_Primary_MetadataXml FOR Value;

Let’s rerun the performance tests with our two queries above, making sure to flush the buffer cache between each execution.

The inevitable happened: the implicit conversion makes it impossible to use the secondary XML index due to a data type mismatch, preventing an actual seek on it. We do see a seek in the second execution plan, but it occurs for every row in the table (500’000 executions) and is essentially just accessing the underlying physical structure stored in the clustered index. In reality, this ‘seek’ is SQL Server scanning the XML to retrieve the exact value of the requested field (in this case, OwnerUserId).
This conversion issue occurs because the function .exist() returns a BIT, while the function .value() returns a SQL type.
This difference in return type can lead to significant performance problems when tuning queries that involve XML.
As explained by Microsoft: “For performance reasons, instead of using the value() method in a predicate to compare with a relational value, use exist() with sql:column()“

Key take-aways

Working with XML in SQL Server can be powerful, but it can quickly become tricky to manage. .exist() and .value() might seem similar, but execution differences and type conversions can have a huge performance impact. Proper XML indexing is essential, and knowing your returned data types can save you from hours of head-scratching. Most importantly, before deciding to store data as XML, consider whether it’s truly necessary ; relational databases are not natively optimized for XML and can introduce complexity and performance challenges.

Sometimes, a simpler and highly effective approach is to extract frequently queried XML fields at the application level and store them in separate columns. This makes them much easier to index and query, reducing overhead while keeping your data accessible.

If your application relies heavily on semi-structured data or large volumes of XML/JSON, it’s worth considering alternative engines. For instance, MongoDB provides native document storage and fast queries on JSON/BSON, while PostgreSQL offers XML and JSONB support with powerful querying functions. Choosing the right tool for the job can simplify your architecture and significantly improve performance.

And to dive even deeper into the topic, with a forthcoming article focused this time on XML storage, keep an eye on the dbi services blogs !

L’article Understanding XML performance pitfalls in SQL Server est apparu en premier sur dbi Blog.