When a company still refuses to put its sensitive data in the cloud, the reason usually isn’t cost or performance: it’s data sovereignty. In the cloud, someone else is the administrator of the machine; therefore the provider can, in theory, read the data files, see the data being used in memory, or read the backups. The threat model is no longer the external attacker but the privileged insider hosting the database.

As covered previously on my blog Beyond TDE and TLS: Bridging the Data Security Governance Gap in Lower Environments, various encryption methods can protect you. For example, TLS protects data in transit and TDE protects it at rest, but as soon as the engine runs a query, it handles plaintext in memory. So there are three states to protect: at-rest, in-transit, in-use. The gap this article focuses on is the last one.

That is exactly what Always Encrypted (SQL Server) and Queryable Encryption (MongoDB) target. The common principle: encryption and decryption happen client-side, in the driver; the keys never reach the engine. The data stays encrypted at rest, in transit, and during processing. The DBA, the cloud operator, the hypervisor admin: all of them see only cyphertext.

That leaves one question: if the engine sees only cyphertext, how does it answer a WHERE condition? Both database engines do answer it, but through different technical means.

SQL Server: queryability lives in the cyphertext

To demonstrate all this, let’s start by creating a table with two columns, Salary and Department, encrypted deterministically on one side and randomized on the other:

DROP TABLE IF EXISTS dbo.Employees;
CREATE TABLE dbo.Employees (
  Id         INT IDENTITY(1,1) PRIMARY KEY,
  LastName   NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  FirstName  NVARCHAR(50) COLLATE Latin1_General_BIN2 NOT NULL,
  DeptDet    NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be DETERMINISTIC
  DeptRand   NVARCHAR(30) COLLATE Latin1_General_BIN2 NOT NULL, -- will be RANDOMIZED
  SalaryDet  INT NOT NULL, -- will be DETERMINISTIC
  SalaryRand INT NOT NULL  -- will be RANDOMIZED
);
GO

INSERT INTO dbo.Employees (LastName, FirstName, DeptDet, DeptRand, SalaryDet, SalaryRand) VALUES
('Martin',  'Alice', 'Sales', 'Sales', 55000, 55000),
('Dubois',  'Bob',   'Sales', 'Sales', 48000, 48000),
('Bernard', 'Chloe', 'Sales', 'Sales', 52000, 52000),
('Petit',   'David', 'IT',    'IT',    72000, 72000),
('Durand',  'Emma',  'IT',    'IT',    68000, 68000);

Five rows: three Sales, two IT. The BIN2 collation is required by Always Encrypted (link to documentation), and the master key lives outside the database (Key Vault, certificate store, or HSM).

Column encryption isn’t done in T-SQL, because the engine doesn’t have the keys. It’s driven from the client (here in PowerShell), declaring for each value a deterministic column and its randomized twin:

Import-Module SqlServer -MinimumVersion 22.0.59
$sqlConnectionString = "Data Source=.\LAB2025;Initial Catalog=AEDEMO;Integrated Security=True;Encrypt=False;Trust Server Certificate=False"
$smoDatabase = Get-SqlDatabase -ConnectionString $sqlConnectionString

$encryptionChanges  = @()
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptDet    -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.DeptRand   -EncryptionType Randomized    -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryDet  -EncryptionType Deterministic -EncryptionKey "CEK1"
$encryptionChanges += New-SqlColumnEncryptionSettings -ColumnName dbo.Employees.SalaryRand -EncryptionType Randomized    -EncryptionKey "CEK1"

Set-SqlColumnEncryption -ColumnEncryptionSettings $encryptionChanges -InputObject $smoDatabase

In this example, I’m working on my 2025 SQL Server instance, on the AEDEMO database, using the column encryption key CEK1 I created beforehand (itself protected by a column master key stored outside the database).

We check that the engine sees the right type per column:

SELECT c.name, c.encryption_type_desc
FROM sys.columns c
WHERE c.object_id = OBJECT_ID('dbo.Employees');

The deterministic mechanism works like this: same plaintext, same cyphertext (an injective function). On an Always Encrypted-enabled connection with Parameterization for Always Encrypted and parameters for the predicates (never literals), equality works:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptDet AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptDet = @d
GROUP BY DeptDet;

The driver encrypts @d with the same key, the server finds the encrypted values that are identical to the parameter, and the result is correct.

You pay for it in three ways…

First weakness: deterministic encryption can cause a data leak. You don’t need the keys to see it. Just connect without Always Encrypted and read the table: the encrypted columns come out as raw binary.

SELECT FirstName, DeptDet, DeptRand FROM dbo.Employees;

Look at the DeptDet column: Alice, Bob, and Chloe share exactly the same blob, and David and Emma share another. Two distinct values across five rows. The adversary doesn’t know what 0x012536… means, but reads the structure: two departments, one with three people, the other with two, and who goes with whom. The DeptRand column, on the other hand, shows five all-different blobs: nothing to read.

It’s harmless on five rows; it isn’t on a real table. On a low-cardinality column (region, sex, status), the distribution of blobs can be compared to a known distribution, and frequency analysis often reconstructs the plaintext. Microsoft’s documentation puts it bluntly: an unauthorized user can guess information by examining patterns, especially when the set of possible values is small.

Second weakness: randomized encryption blocks every query. With randomized encryption, the driver adds a fresh random value to each cell before encrypting, so the same input produces a different cyphertext every time. Therefore, the leak is gone but so is the query:

DECLARE @d NVARCHAR(30) = 'Sales';
SELECT DeptRand AS Enc, COUNT(*) FROM dbo.Employees WHERE DeptRand = @d
GROUP BY DeptRand;  

Msg 33277, Level 16, State 2, Line 6
Encryption scheme mismatch for columns/variables ‘DeptRand’, ‘@d’. The
encryption scheme for the columns/variables is (encryption_type =
‘RANDOMIZED’, …) and the expression near line ‘6’ expects it to be
DETERMINISTIC, or RANDOMIZED, a BIN2 collation for string data types,
and an enclave-enabled column encryption key, or PLAINTEXT.

The server can no longer compare, since the same plaintext produces a different cyphertext on every row. We gained confidentiality and lost the query. It’s all or nothing.

Third weakness: no range, even with deterministic. Byte equality says nothing about order:

DECLARE @s INT = 55000;
SELECT FirstName FROM dbo.Employees WHERE SalaryDet < @s;

WHERE SalaryDet < @s fails even though the column is deterministic. Sorting, BETWEEN, LIKE: out of reach for Always Encrypted alone.

The verdict is clear: bare Always Encrypted means equality, or nothing (=, IN, GROUP BY, and DISTINCT supported).

MongoDB: queryability lives in a protocol

MongoDB’s Queryable Encryption makes optimal use of randomized encryption. On the server side, everything is Randomized-encrypted, as the documentation explains: the server has no knowledge of the data it processes. The encrypted view (a client connected without the keys) confirms it: even the three Sales employees come out with all-different BinData.

No frequency leak, unlike SQL Server’s deterministic encryption. With the keys, the same find returns the plaintext:

The technical mechanism behind this lies in the collection’s declaration. Each encrypted field carries a queryType, its parameters, and a distinct key (keyId), one Data Encryption Key per field, which is mandatory:

And queries with equality tests, range, and even equality on a field declared as range all work:

On the server side, MongoDB then maintains encrypted index structures, and for each query the driver generates cryptographic tokens that the server checks against those structures without ever seeing the plaintext. This is a searchable encryption scheme. Range is available in GA, with no special hardware.

The real difference comes down to one thing: the driver

On the SQL Server side, the driver’s work stays thin: it encrypts the parameters, rewrites the query, decrypts the results. Queryability itself is already carried by the cyphertext; the driver doesn’t have to handle it. On the MongoDB side, the driver carries the whole protocol: it generates the tokens checked against the encrypted indexes.

Once the collection is in place, find({ department: "Sales" }) is written like a normal query and the driver handles the encryption on its own. Each encrypted field needs its own key (Data Encryption Key). The master key must be pinned, otherwise orphaned keys return an HMAC validation failure error. And the encryption API is only available from a client created as encrypted, so not from a standard Compass connection, for example.

Summary

What if you wanted range while staying on SQL Server?

This is where secure enclaves come in. The engine delegates the computation to an enclave: a protected memory region where the data is decrypted and processed in the clear, out of reach, including from the machine’s administrator. This is what unlocks range, LIKE, sorting, and in-place encryption: inside the enclave the server no longer compares encrypted bytes, it works on the plaintext. This gain has a price. It requires compatible hardware or secure virtualization, an attestation service to deploy and maintain, and keys configured for the enclave, which noticeably increases architectural complexity compared with classic Always Encrypted.

But it also changes the nature of the trust. Queryable Encryption rests on a cryptographic guarantee: the server cannot read, it’s a mathematical property. Enclaves rest on a hardware guarantee: you trust the CPU, and its attestation, to isolate the protected region.

There remains a third path, often mentioned: homomorphic encryption (FHE), which computes directly on the cyphertext without ever decrypting it. Elegant on paper, but out of the game for database search: the computational cost is massive and response time collapses as the volume grows. So the practical choice really does play out between the two worlds described here.

L’article SQL Server vs MongoDB: When the cloud is your adversary (Always Encrypted vs Queryable Encryption) est apparu en premier sur dbi Blog.

This is something we often see among our customers. If we assume that we carry out a migration project roughly every five years, this means we’re keeping databases that are taking up space for no reason.

We, however, are Green SQL Server DBA; we’ll look to manage it correctly.

The AUDIT

First of all, what do we need to analyse?

This does not apply to all databases, but only to those that are obsolete.

They can be categorised into two types:

• Databases Never Accessed
  • No application connections detected
  • No recent transactions or queries after a period (3 months for exemple)
• Abandoned Databases
  • Linked to cancelled projects
  • No activity and nobody tell it

To see databases never accessed and abandoned databases I use the DMV sys.dm_db_index_usage_stats to see if I have an access to the database (read or write):

SELECT d.name AS DatabaseName, MAX(ius.last_user_seek) AS LastSeek,MAX(ius.last_user_scan) AS LastScan,
    MAX(ius.last_user_lookup) AS LastLookup,

    (
        SELECT MAX(v.ActivityDate)
        FROM (VALUES
                (MAX(ius.last_user_seek)),
                (MAX(ius.last_user_scan)),
                (MAX(ius.last_user_lookup))
             ) v(ActivityDate)
    ) AS LastReadActivity,

    MAX(ius.last_user_update) AS LastWriteActivity,

    DATEDIFF(DAY,
        (
            SELECT MAX(v.ActivityDate)
            FROM (VALUES
                    (MAX(ius.last_user_seek)),
                    (MAX(ius.last_user_scan)),
                    (MAX(ius.last_user_lookup))
                 ) v(ActivityDate)
        ),
        GETDATE()
    ) AS DaysSinceLastRead,

    DATEDIFF(DAY,
        MAX(ius.last_user_update),
        GETDATE()
    ) AS DaysSinceLastWrite

FROM sys.databases d
LEFT JOIN sys.dm_db_index_usage_stats ius
    ON d.database_id = ius.database_id
WHERE d.database_id > 4  -- Exclude system databases
GROUP BY d.name
ORDER BY LastWriteActivity ASC;

We can see on the first databases in the list, that we have null everywhere… This means that the databases were not solicited since the last restart… Good candidate for unused databases!

This brings us to the next point.

The ANALISYS

If we can see that there is little activity on the database, we can, of course, carry out an initial analysis using criteria such as these:

< 30 days –> the database is still active

30–180 days –> the database is on hold pending to be dropped

> 180 days –> the database is a strong candidate to be dropped

Of course, it also depends on your environment – whether it’s dev, test or prod – and the criteria will vary accordingly.

The IMPACT

Like in my precedent post, we will have same impacts but not for queries of courses!

The first impact is the storage and the backup

If the database represents ~100 GB on the disk , it’s also 100 GB backed up unnecessarily.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 1 TB!

The third impact is on the maintenance plan

The following operations must also handle the unused database:

– CheckDB
– Backups

As the database is not used, the index rebuild/reorg & update Stats will be fast!

Each unused database extends the maintenance windows for the checkdb and backups…

The ADVISE

The first advice would be to plan well in advance.

Indeed, when creating or restoring a database, it is important to know its lifecycle.

Don’t hesitate to challenge the application owners on a deadline and make sure to add it to your calendar +1day.

The second advice is to set up a monthly monitoring process prior to Windows or SQL Server patch cycles so that you can identify these unnecessary databases.

And before dropping a database, I advice you to put the database offline and see if after some days somebody complaints…

In the majority of cases, you will not have reactions, believe me!

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will let the unused databases until the next migration.

A Green SQL Server DBA will check it periodically and manage it.

An unused database is like a car that never leaves the garage.
 It still costs money, requires maintenance, and consumes resources.
The Green SQL Server DBA notices it before it becomes waste…

Think about it and begin now to see to monitor your unused databases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #3 – The hidden cost of forgotten databases est apparu en premier sur dbi Blog.

Can this also be applied to PostgreSQL?

In short, yes, it is possible. Let’s see how.

Here is the list of the previous blog posts:

• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-2-3/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-rest-api-with-sql-server-2025-3-3/

Partitioning and filesystem

We will reuse the sqlpool ZFS pool created in the first part of this series.

We identify the 300 GB disk attached to the VM. In our case, it is /dev/sdb, backed by the sqlpool/pve/vm-307-disk-0 zvol on the Proxmox side:

lsblk

We create a single partition of type Linux filesystem:

sudo sgdisk -n 1:0:0 -t 1:8300 /dev/sdb

We format the partition with XFS, which is the most commonly recommended filesystem for PostgreSQL data directories:

sudo mkfs.xfs -L pgdata /dev/sdb1 -f

We verify the result:

ahi@pgl:~$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="pgdata" UUID="028afa2f-7bb3-4a40-92aa-91c1a33f18ae9" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="bd54f285-092e-4b1b-ba5e-6877f054fa7"

Mountpoint:

We create the mount point:

sudo mkdir -p /pgdata

Persistent mount via fstab:

We add the mount entry to /etc/fstab using the filesystem label rather than the device name. The device name (/dev/sdb) may change if disks are added or removed while the label remains stable:

echo 'LABEL=pgdata /pgdata xfs noatime,nodiratime 0 2' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /pgdata

We verify that the volume is mounted:

df -h /pgdata

PostgreSQL installation

We install PostgreSQL 18 from the official PGDG repository, which provides the latest PostgreSQL versions for Ubuntu:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt install -y postgresql-18

Cluster creation on /pgdata:

The Ubuntu packages create a default cluster under /var/lib/postgresql. This is not what we want. The data files and the WAL must both reside on the ZFS-backed volume, so that a single ZFS snapshot captures a consistent state of the database. If they were on different volumes, the snapshot would not be atomic.

We drop the default cluster and recreate it on /pgdata:

ahi@pgl:~$ sudo pg_dropcluster --stop 18 main
sudo install -d -o postgres -g postgres -m 700 /pgdata/18
sudo pg_createcluster -d /pgdata/18/main 18 main
sudo systemctl enable --now postgresql@18-main
Creating new PostgreSQL cluster 18/main ...
/usr/lib/postgresql/18/bin/initdb -D /pgdata/18/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /pgdata/18/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
             Ver Cluster Port Status Owner     Data directory  Log file
             18  main    5432 down   postgres  /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql@18-main.service → /usr/lib/systemd/system/postgresql@.service.

We verify that the cluster is online and located on the right volume:

ahi@pgl:~$ pg_lsclusters
Ver Cluster Port Status Owner     Data directory  Log file
18  main    5432 online <unknown> /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
ahi@pgl:~$ sudo -u postgres psql -c "SHOW data_directory;"
 data_directory
-----------------
 /pgdata/18/main
(1 row)

ahi@pgl:~$ sudo -u postgres psql -c "SELECT version();"
                                                             version
-------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
(1 row)

We can also confirm that the WAL directory lives inside the data directory, and therefore on the zvol:

ls -ld /pgdata/18/main/pg_wal

Creating a large database

We need a database large enough to make traditional backup and restore operations time-consuming. In the SQL Server part of this series, we used the StackOverflow database (about 207 GB). For PostgreSQL, we use pgbench, the benchmarking tool shipped with PostgreSQL.

We create the database and initialize it with a scale factor of 10000. This produces a database of approximately 146 GB, with 1 billion rows in the pgbench_accounts table:

sudo -u postgres createdb bench
sudo -u postgres pgbench -i -s 10000 --partitions=8 bench

A few minutes later:

We can monitor the data growth during the initialization:

watch -n 30 'df -h /pgdata'

A few minutes later:

On the Proxmox side:

After some time, the process completes:

vacuuming...

creating primary keys...
done in 1559.55 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 598.12 s, vacuum 675.53 s, primary keys 285.88 s).
ahi@pgl:~$

We check the database size:

We run a checkpoint before taking the snapshot. The recovery process starts replaying the WAL from the last checkpoint. By running it right before the snapshot, almost nothing needs to be replayed when the database starts after a restore:

sudo -u postgres psql -c "CHECKPOINT;"

Comparison with SQL Server:

On the SQL Server side, we had to run SUSPEND_FOR_SNAPSHOT_BACKUP and BACKUP WITH METADATA_ONLY. On the PostgreSQL side, none of that is needed.

The data files and the WAL are on the same zvol. An atomic ZFS snapshot therefore captures a state equivalent to a power loss, and PostgreSQL is designed to recover cleanly from that state through crash recovery: the WAL is replayed from the last checkpoint. This is documented and officially supported.

The snapshot is the backup. There is no .bkm file, no metadata backup.

Snapshot process flow

On the Proxmox side, we create the snapshot and protect it with a hold:

SNAP="sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)" 
zfs snapshot "$SNAP" 
zfs hold sqlsnap "$SNAP" 
echo "$SNAP"

The hold protects the snapshot from an accidental destruction, as we did in part 2 of this series. We note the exact snapshot name, it will be needed for the restore.

The database stays online during the whole operation. No I/O freeze is required.

We list the snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We drop the database then we restore the snapshot:

We run the snapshot restore procedure. On the VM, we stop the cluster and unmount the volume:

sudo systemctl stop postgresql@18-main
sudo umount /pgdata

On the Proxmox side, we want to restore our snapshot. We can list the available snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We roll back the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T165409

On the VM, we mount the volume and start the service:

sudo mount /pgdata
sudo systemctl start postgresql@18-main

We check a few elements in the logs:

sudo tail -30 /var/log/postgresql/postgresql-18-main.log

The service shutdown, then the restart:

UTC [232108] LOG:  database system is shut down
UTC [233968] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
UTC [233968] LOG:  listening on IPv4 address "0.0.0.0", port 5432
UTC [233968] LOG:  listening on IPv6 address "::", port 5432
UTC [233968] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

PostgreSQL detects that the database was not shut down properly and replays the WAL. This is the same crash recovery mechanism as in SQL Server. Finally, the database starts.

UTC [233974] LOG:  database system was not properly shut down; automatic recovery in progress
UTC [233974] LOG:  redo starts at 20/2CB76278
UTC [233974] LOG:  invalid record length at 20/2CB76380: expected at least 24, got 0
UTC [233974] LOG:  redo done at 20/2CB76348 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
UTC [233974] LOG:  checkpoint starting: end-of-recovery immediate wait
UTC [233972] LOG:  checkpoint complete: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.009 s, total=0.030 s; sync files=0, longest=0.000 s, average=0.005 s; distance=0 kB, estimate=0 kB; lsn=20/2CB76380, redo lsn=20/2CB76380
UTC [233968] LOG:  database system is ready to accept connections

We then verify that the database is available again:

sudo -u postgres psql -d bench -c "SELECT pg_size_pretty(pg_database_size('bench'));"

Consistency proof under load

The previous test was done on a quiesced database: we ran a CHECKPOINT right before the snapshot, and nothing was writing. The real question is: what happens if the snapshot is taken while the database is being written to?

This is where PostgreSQL differs the most from SQL Server. There is no SUSPEND_FOR_SNAPSHOT_BACKUP. We take the snapshot in the middle of the write activity and we let the WAL replay do the work.

We start the load. The built-in pgbench script runs a TPC-B-like transaction: three UPDATE statements on the accounts, tellers and branches tables and one INSERT into the history table:

sudo -u postgres pgbench -c 8 -j 4 -T 300 bench &

While the load is running, we take a snapshot on the Proxmox side:

zfs snapshot sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)

zfs hold sqlsnap sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551

No CHECKPOINT this time, no freeze. The database is actively writing while the snapshot is taken.

The state after some time under load:

We stop the service:

sudo systemctl stop postgresql@18-main 
sudo umount /pgdata

We restore the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551 

We mount the volume, start the service and check the logs:

sudo mount /pgdata 
sudo systemctl start postgresql@18-main

This time the log shows a real recovery:

2026-08-03 20:49:03.244 UTC [234405] LOG:  database system is shut down
2026-08-03 20:50:34.333 UTC [234620] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv6 address "::", port 5432
2026-08-03 20:50:34.335 UTC [234620] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2026-08-03 20:50:34.343 UTC [234626] LOG:  database system was interrupted; last known up at 2026-08-03 20:44:36 UTC
2026-08-03 20:50:34.381 UTC [234626] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-03 20:50:34.384 UTC [234626] LOG:  redo starts at 20/7A3618B0
2026-08-03 20:50:38.024 UTC [234626] LOG:  invalid record length at 20/8BC0A4F8: expected at least 24, got 0
2026-08-03 20:50:38.024 UTC [234626] LOG:  redo done at 20/8BC0A4D0 system usage: CPU: user: 0.71 s, system: 0.63 s, elapsed: 3.63 s
2026-08-03 20:50:38.028 UTC [234624] LOG:  checkpoint starting: end-of-recovery immediate wait
2026-08-03 20:50:53.936 UTC [234624] LOG:  checkpoint complete: wrote 105355 buffers (53.6%), wrote 5 SLRU buffers; 0 WAL file(s) added, 17 removed, 0 recycled; write=3.648 s, sync=12.232 s, total=15.911 s; sync files=189, longest=12.223 s, average=0.065 s; distance=287395 kB, estimate=287395 kB; lsn=20/8BC0A4F8, redo lsn=20/8BC0A4F8
2026-08-03 20:50:53.948 UTC [234620] LOG:  database system is ready to accept connections

Three differences compared to the first test:

• The “last known up at” timestamp (20:44:36) does not match a checkpoint we ran manually. It matches the last automatic checkpoint triggered during the load.

• The redo is not instantaneous anymore: 3.63 seconds, replaying about 280 MB of WAL (from LSN 20/7A3618B0 to 20/8BC0A4D0). All the write activity between the last checkpoint and the snapshot had to be replayed. The transactions committed before the snapshot are recovered, the ones that were in flight are rolled back.

• The end-of-recovery checkpoint then writes everything the redo rebuilt in memory: 105355 buffers, 53.6% of the buffer pool. The database is ready to accept connections about 19 seconds after the service start.

The crash recovery completed correctly and the database has been restored. We verify the TPC-B invariant. Each pgbench transaction applies the same delta to the accounts, tellers and branches tables in a single transaction. On a consistent database, the three sums must be equal:

Major drawbacks

• The snapshot covers the whole zvol. All the databases of the cluster are captured and restored together. There is no per-database restore, unlike the METADATA_ONLY approach on SQL Server which targets a single database.
• There is no backup history. SQL Server records the metadata backup in msdb. Here, the only trace is the snapshot itself on the ZFS side.
• Point-in-time recovery is not covered. The snapshot alone brings the database back to the moment it was taken. For PITR, WAL archiving would still be required on top of it.

Conclusion

• The snapshot backup and restore model of the SQL Server series applies to PostgreSQL (no I/O freeze, no metadata backup).
• One important rule: data files and WAL must reside on the same zvol so the snapshot is atomic.
• A 146 GB database was restored in a few seconds and in less than 20 seconds under active load, WAL replay included.

Thank you. Amine Haloui

L’article PostgreSQL Snapshot Backup and Restore with Proxmox ZFS (4/4) est apparu en premier sur dbi Blog.

After creating the Ticket, I connect to the instance and see that the msdb system database is in a Suspect State:

After taking a deep breath because I don’t see that every day, I begin my investigation by asking the state with the system view sys.databases first:

msdb is really in a suspect mode…

Now, to begin the analysis, I go to read the error log.:

The Error Log give me the error that I comments in SSMS query to read it better:

Error: 3314, Severity: 21, State: 1.
During undoing of a logged operation in database ‘msdb’, an error occurred at log record ID (821931:128397:5).
Typically, the specific failure is logged previously as an error in the Windows Event Log service. Restore the database or file from a backup, or repair the database.

If you have this error message, the best is to restore the msdb;
As the information of the backups is in the msdb, I cannot do a right-click to restore the database…  

I looked for the latest backup in the error log and find it:

But I also sse that the checkdb before each backup give 12 errors…

My question is at this point: Wasn’t there an alert saying that the database was corrupted?

As the users are blocked, I need to react quickly.
This is why, I do my restore and a CHECKDB with  REPAIR_ALLOW_DATA_LOSS:

RESTORE DATABASE msdb FROM DISK = 'xxx\msdb\msdb_fullbackup_xxxx20260801xxxx.BAK' WITH REPLACE;
GO
 ALTER DATABASE msdb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
GO
DBCC CHECKDB ('msdb', REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS, ALL_ERRORMSGS;
GO
ALTER DATABASE msdb SET MULTI_USER;

The commands completed successfully! YES!

The database msdb is online and working fine. SUPER!

To be sure, I go in the errorlog to see the restore and the checkdb with REPAIR_ALLOW_DATA_LOSS:

The restore was successful and in the checkDB with can see 12 errors and repaired 12 errors.

If I will investigate deeper, I will go the dump file and analyse it.

I finally managed to sort out and resolve the critical alert on monday morning.
Users can connect and read their tables again…
The customer knows why he comes to us for an SLA…

A good victory for a monday morning in our support SLA dbi-services!
See you soon for sharing again customer cases!

L’article SQL Server:  msdb is in Suspect State est apparu en premier sur dbi Blog.

We turn up on a Monday morning and there are alerts from the weekend saying

[MSSQL] – Instance Dark_Vador-Data(F:): Disk space in GB on server Death_Star is critical

Info: NOT OK – Free space: 20 MB out of 200 GB

The storage is almost full and:
– no projects were deployed over the weekend
– no exceptional data loads were scheduled
– nobody works at the weekend

A strange phenomenon to come across on a Monday morning….

On closer inspection, the findings are surprising; we can see that a database within the instance has been undergoing regular autogrowth for weeks…

Every day, there are several regular autogrowth on the data file for this database until the saturation of the disk.

Who is the culprit?

Not the business data, but one or more tables named xxxxLog or Historyxxx, or the combination of both HistoryLog!

PS: It’s an example but in the reality before the weekend, we will have Warnings and in the majority of the cases we react before the critical alert of course!

Data created to diagnose a problem, track an activity or maintain a history, which is then never cleaned up or has a default retention period that is too long and unsuitable for the context, is not always easy to detect.

On the other hand, it’s always easier to expand disk capacity without too much effort, and everything runs smoothly in an ideal world…

We, however, are Green SQL Server DBA; we’ll look deeper.
Our curiosity will lead us to challenge the business on these points.

The AUDIT

First, you need to know the autogrowth settings for these databases and how the data file growth.
Here is a simple script that shows the settings:

SELECT
    DB_NAME(database_id) AS DatabaseName,
    name AS LogicalFileName,
    type_desc AS FileType,
    physical_name AS PhysicalFileName,
    size / 128.0 AS CurrentSizeMB,
    CASE
        WHEN is_percent_growth = 1
            THEN CAST(growth AS VARCHAR(10)) + ' %'
        ELSE CAST(growth / 128 AS VARCHAR(10)) + ' MB'
    END AS AutoGrowth,
    max_size,
    CASE
        WHEN max_size = -1 THEN 'Unlimited'
        ELSE CAST(max_size / 128 AS VARCHAR(20)) + ' MB'
    END AS MaxSize
FROM sys.master_files where type_desc='ROWS' AND database_id > 4
ORDER BY
    DatabaseName,
    FileType,
    LogicalFileName;

Like for my precedent blog “Green SQL Server DBA Tips  #1 – Are unnecessary indexes cluttering up your database?”, I use the Dynamics database as example:

In my script, I exclude all system databases and the Log File to be concentrated to the Data File.

Now, I need to have the data file growth. This is through the default trace file and the event 92:

DECLARE @TraceFile NVARCHAR(500);

SELECT @TraceFile = path
FROM sys.traces
WHERE is_default = 1;

IF @TraceFile IS NULL
BEGIN
    RAISERROR('Default Trace is disabled.',16,1);
    RETURN;
END

;WITH GrowthEvents AS
(
    SELECT
        DB_NAME(DatabaseID) AS DatabaseName, FileName, CAST(StartTime AS DATE) AS GrowthDate,
        DATEPART(YEAR, StartTime) AS YearNum, DATEPART(WEEK, StartTime) AS WeekNum,
        DATEPART(MONTH, StartTime) AS MonthNum, IntegerData * 8.0 / 1024 AS GrowthMB
    FROM fn_trace_gettable(@TraceFile, DEFAULT)
    WHERE EventClass IN (92) -- 92 is data growth
)

SELECT DatabaseName,FileName,GrowthDate,COUNT(*) AS GrowthEvents,SUM(GrowthMB) AS TotalGrowthMB
FROM GrowthEvents GROUP BY DatabaseName,FileName,GrowthDate ORDER BY DatabaseName, GrowthDate

As I don’t have any growth on the dynamicsBC database, I switch to the M-Files database for my example:

We have here some daily growth…

Go to see if we have historical or log tables in M-Files with this simple script who give us  the table and the row count and MB:

SELECT s.name AS Schema_Name,t.name AS Table_Name,
    SUM(p.rows) AS Row_Count, CAST(SUM(a.total_pages) * 8.0 / 1024 AS DECIMAL(18,2)) AS SizeMB
FROM sys.tables t INNER JOIN sys.schemas s     ON t.schema_id = s.schema_id
INNER JOIN sys.indexes i ON t.object_id = i.object_id
INNER JOIN sys.partitions p ON i.object_id = p.object_id  AND i.index_id = p.index_id
INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id
WHERE t.name LIKE 'Histor%' OR t.name LIKE '%Log'
GROUP BY s.name, t.name ORDER BY SizeMB DESC;

The ANALISYS

In my case, we can see that we have 383550 lignes (19.73Mb) on the table named OBJECTTYPECHANGELOG for a M-Files database.

It’s only ~20MB here but in few months perhaps 20 GB or more…

As a Green SQL Server DBA, I will contact the owner of M-Files project otherwise don’t forget that dbi services has also the expertise in ECM tools like M-Files.

The IMPACT

Like in my precedent post, we will have the same impacts

The first impact is on DML INSERT operations.

With every insertion, the log or historical information is written to the table and update perhaps also index and at the end is never used by a query.

The second impact is the storage and the backup

If the log table represents ~20 GB of the database.

It’s 20 GB backed up unnecessarily, 20 GB restored unnecessarily and, above all, 20 GB stored unnecessarily and just for one environment.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 100 GB!

The third impact is on the maintenance plan

The following operations must also handle the log/historical tables:
– Rebuild Index
– Reorganise Index
– CheckDB
– Backups
– Restores

Each growth of log/historical tables extends the maintenance windows…

The ADVISE

You really need to be careful before deleting anything in these table.

Don’t Truncate the table because you see it!

First see with the vendor if a purge or a retention parameter exists through the application. 

In my case with M-Files and the table OBJECTTYPECHANGELOG, in the administration interface, you can configure the retention. Go to challenge your M-Files team on it!

Finally, for every large log or historical table, you need to ask yourself the following questions:
– Who uses it?
– How often?
– Is there a legal obligation?
– What retention period is required?
– Can it be archived?
– Can it be purged?

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will see a data disk full and says:

“How much GB need we to extend the disk?”

A Green SQL Server DBA sees a data disk full and says:

“Why is my disk full and what can I do?”

Just like adding a larger trunk doesn’t make a car more efficient, adding more GB doesn’t make a database healthier.

The greenest SQL Server DBA is not the one with the largest disks but it’s the one that keeps only data that continues to deliver value.

Think about it and begin now to see if you have cases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #2 – When logs and history records cause your database to growth and crash est apparu en premier sur dbi Blog.

Logical fragmentation was a problem on spinning disks: an ordered scan on a fragmented index broke the read-ahead into smaller I/Os and each jump cost a disk head movement. On SSD, NVMe and cloud storage, the penalty of non-sequential reads is marginal. And the “physical order” we used to restore so carefully is only the order of the pages inside the data file: below it, the file system, the SSD controller and the storage layer place the blocks wherever they want anyway.

Here is the mapping to remember

• Internal fragmentation = page density = fixed by automatic index compaction
• External fragmentation = page order = ignored by automatic index compaction (it can even increase and we will see it in the demo)

How does it work ?

Automatic index compaction is not a hidden maintenance job. It is an additional task performed by a background process that already exists: the PVS cleaner is a component of Accelerated Database Recovery (ADR).

Here is the principle

• The PVS cleaner periodically visits the pages that were recently modified (insert, update, delete) to remove obsolete row versions
• When automatic index compaction is enabled the cleaner also checks if the visited page has free space excluding the space reserved by the fill factor
• If so, it moves rows from the next page into the current page as long as they fit and repeats the operation on a few consecutive page pairs
• A page that becomes empty is deallocated

The result: the number of pages decreases, the page density increases, and the storage space, I/O, CPU and buffer pool consumption decrease.

The overhead is minimal because the process only considers recently modified pages, unlike a rebuild or a reorganize which process all the pages. Like a reorganize, the compaction acquires short-term exclusive page locks to move the rows. If a lock cannot be acquired immediately, the page is simply skipped and will be considered again later.

One command per database, no restart, no exclusive access. The compaction starts within minutes:

-- Enable the feature
ALTER DATABASE [SQL-DB-1] SET AUTOMATIC_INDEX_COMPACTION = ON;

-- Check if it's enabled
SELECT name, is_automatic_index_compaction_on FROM sys.databases;

One important point before you enable it

The compaction process only considers the pages modified after you enable the feature. If the page density of your indexes is already low, run a one-time reorganize or rebuild to fix the existing situation. From that point on, the automatic compaction keeps the indexes compact without any action on your side.

Demo

• We create a table with a clustered primary key and we insert 5 million rows
• We measure the baseline: page count, page density, fragmentation
• We delete 2 rows out of 3, scattered over the whole table, to simulate index bloat
• We measure again and we let the engine work

We create the table and we insert 50K rows:

CREATE TABLE dbo.Demo (
    Id      int IDENTITY CONSTRAINT PK_Demo PRIMARY KEY CLUSTERED,
    Payload char(400) NOT NULL DEFAULT REPLICATE('X', 400)
);

INSERT INTO dbo.Demo (Payload)
SELECT TOP (50000) REPLICATE('X', 400)
FROM sys.all_columns a CROSS JOIN sys.all_columns b;

We measure the baseline:

SELECT index_level, page_count, avg_page_space_used_in_percent,
       avg_fragmentation_in_percent, record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Demo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

We measure the baseline:

SELECT index_level, page_count, avg_page_space_used_in_percent,
       avg_fragmentation_in_percent, record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.Demo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

The internal fragmentation is low.

Here is what we have:

However to observe the automatic index compaction in action we need to create the following situation:

Here is the baseline:

We create the bloat

We delete 2 rows out of 3 scattered uniformly over the table. No page becomes empty. Every page keeps around one third of its rows. We change page density:

DELETE FROM dbo.Demo WHERE Id % 3 <> 0;

Without automatic index compaction, the expectation is simple: the page count stays at 277778 (deletes alone do not deallocate non-empty pages) and the density drops to around 31% (one third of 94.93%).

We measure again

We let the engine finish its job and we check again

The engine did the work in the background while the database stayed fully available.

This feature replaces the compaction part of your maintenance strategy not the whole strategy.

A summary:

• It does not update the statistics. A rebuild updates them the automatic compaction does not. If you rely on your rebuild jobs to refresh the statistics, keep a statistics update job.
• It does not reduce the logical fragmentation and it can even increase as we saw in the demo. For most workloads this has no measurable impact
• It does not recreate the free space reserved by the fill factor. Only a rebuild does. Workloads that need a fill factor below 100 to reduce page splits can still benefit from an occasional rebuild
• It does not shrink the data files. The used space decreases, the allocated size does not change
• It only applies to the leaf level of B-tree indexes in IN_ROW_DATA allocation units. Heaps, LOB data, row-overflow data, compressed columnstore rowgroups and memory-optimized tables are not concerned
• Indexes with page locks disabled (ALLOW_PAGE_LOCKS = OFF) are not eligible

Reference

Automatic Index Compaction – SQL Server | Microsoft Learn

Thank you. Amine Haloui

L’article SQL Server: Automatic index compaction in Azure (preview) – Part 2 est apparu en premier sur dbi Blog.

Microsoft recently released a new feature in public preview: automatic index compaction. The database engine takes care of compacting the indexes itself, in the background, while the data changes. No job, no maintenance window, no script to maintain.

Here is what we will cover in this blog post

• Which problems this feature addresses
• The requirements
• The difference between page density and index fragmentation (and why it matters)
• What the feature does not do

This is the first part of a series of two blog posts.

Which problems does it address?

Automatic index compaction is Microsoft’s answer to a routine operational task: index maintenance jobs.

Here are the problems with these jobs

• They are expensive in CPU and I/O. A rebuild reads and rewrites the entire index
• They take time to run, and maintenance windows keep shrinking
• They generate a lot of transaction log, which impacts log backups and Always On replication
• They can block applications. An offline rebuild locks the table and even an online rebuild needs a short exclusive lock at the beginning and at the end
• Someone has to set them up, monitor them and fix them when they fail

There is also a less obvious problem: for years, these jobs have been optimizing the wrong metric. We will come back to this point in a moment.

Requirements

The feature is currently in public preview and available on

• Azure SQL Database
• Azure SQL Managed Instance, with the Always-up-to-date update policy
• SQL database in Microsoft Fabric

It is not available on SQL Server on-premises, including SQL Server 2025.

There is nothing to install. The feature relies on a background process that already runs on these platforms (the ADR cleaner, more on this below), and Accelerated Database Recovery is always enabled on Azure SQL Database, Azure SQL Managed Instance and SQL database in Fabric.

Internal and external fragmentation

When we talk about index fragmentation, we are actually talking about two different phenomena. They are measured by two different columns of sys.dm_db_index_physical_stats, they have different causes, and above all they have very different costs.

Before we start: how is a data file organized ?

To understand fragmentation, we need one prerequisite: how SQL Server sees its data files.

• For any program, a file is an array of bytes provided by the operating system. Byte 0, byte 1, byte 2, and so on. Reading means asking for a range: an offset and a length

• SQL Server slices this array into pages of 8 KB. The page N is, by definition, the bytes N × 8192 to (N+1) × 8192 − 1. The page ID is not a label, it is an address in the file

• Everything below the file (NTFS clusters, SSD, SAN volumes) is translated and hidden by the storage stack.

Every time we say “physical order” in the rest of this post, we mean the order of the pages inside the data file not their placement on the hardware.

What does it look like?

Internal fragmentation: the pages are not full

Internal fragmentation means that the pages of an index contain free space. It is measured by avg_page_space_used_in_percent, also called page density.

Where does it come from?

• Deletes, especially scattered deletes: the rows disappear, the pages stay
• Page splits: when a page is full, an insert in the middle of the key range or an update that makes a row grow moves half of the rows to a newly allocated page. Both pages end up around 50% full
• Updates that shrink variable length columns
• Large rows that do not pack well: a 5000 bytes row means one row per page, and around 38% of every page is lost mechanically

Note that some free space is normal, and even intentional: the space reserved by the fill factor is internal fragmentation on purpose to absorb future inserts without page splits.

What does it look like?

CREATE DATABASE DemoFrag
GO

ALTER DATABASE DemoFrag SET RECOVERY SIMPLE
GO

USE DemoFrag
GO

CREATE TABLE dbo.SplitDemo (
    Id      int NOT NULL CONSTRAINT PK_SplitDemo PRIMARY KEY CLUSTERED,
    Payload varchar(8000) NOT NULL
);

-- 2 lines about 4000 bytes per page
INSERT INTO dbo.SplitDemo VALUES
(1, REPLICATE('A', 4000)), (2, REPLICATE('B', 4000));

My page is almost full:

Why does it cost ?

Because the same rows are spread over more pages than necessary and everything in SQL Server is done at the page level:

• The buffer pool caches pages, not rows. A page at 40% density wastes 60% of the memory it occupies
• Every read, logical or physical, handles more pages for the same data
• Backups are bigger, integrity checks are longer

This is the key point: internal fragmentation follows the data everywhere, including in RAM. It is a permanent tax on memory, I/O and CPU. However it’s not the case with external fragmentation.

External fragmentation: the offsets do not follow the logical order

At the leaf level of a B-tree, the pages are chained together in the order of the index keys with two pointers stored in every page header (m_nextPage, m_prevPage). This chain is the logical order and it cannot be wrong: it is the order of the keys by definition.

External fragmentation (also called logical fragmentation) means that following this chain no longer corresponds to reading the file sequentially. The pages are logically ordered, but their offsets in the file are not. It takes two forms:

• Holes: the pages of the index are not at consecutive offsets. Pages of other objects, or free pages, sit in between
• Disorder: the pages are at consecutive offsets, but chained in a different order

avg_fragmentation_in_percent counts both forms without distinction. The columns fragment_count and avg_fragment_size_in_pages are closer to the real cost: they measure the length of the contiguous runs.

Where does it come from?

• Several objects growing at the same time: the engine allocates the extents in the order the requests arrive, so the objects interleave in the file. This is the normal life of a database
• Page splits: the new page is allocated where free space is found, rarely next to the original page
• Page deallocations: massive deletes, and the automatic index compaction itself

What does it look like?

USE DemoFrag
GO

CREATE TABLE dbo.SplitDemo (
    Id      int NOT NULL CONSTRAINT PK_SplitDemo PRIMARY KEY CLUSTERED,
    Payload varchar(8000) NOT NULL
);

-- 2 lines about 4000 bytes per page : lines 1 and 2 on 1 page, 3-4 on next page

INSERT INTO dbo.SplitDemo VALUES
(1, REPLICATE('A', 4000)), (2, REPLICATE('B', 4000)),
(3, REPLICATE('C', 4000)), (4, REPLICATE('D', 4000))

-- Lines location (file:page:slot)?
SELECT Id, sys.fn_PhysLocFormatter(%%physloc%%) AS physical_location
FROM dbo.SplitDemo;

My first 2 rows are located on page 376 and the last 2 rows on page 378.

What does page 377 contain? I thought the pages were contiguous?

Since my table has a clustered index on Id it is stored as a B-tree. Can we see it?

SELECT allocated_page_page_id, page_type_desc, page_level, previous_page_page_id, next_page_page_id
FROM sys.dm_db_database_page_allocations(DB_ID(), OBJECT_ID('dbo.SplitDemo'),1, NULL, 'DETAILED');

What does it look like internally?

SELECT * FROM sys.dm_db_page_info (10, 1, 376, DEFAULT);

SELECT * FROM sys.dm_db_page_info (10, 1, 378, DEFAULT);

Page split to generate external fragmentation:

-- Split happens
UPDATE dbo.SplitDemo SET Payload = REPLICATE('B', 4100) WHERE Id = 2;

-- Check page density
SELECT index_level, page_count, avg_page_space_used_in_percent,
avg_fragmentation_in_percent, record_count,
ghost_record_count, version_ghost_record_count
FROM sys.dm_db_index_physical_stats(DB_ID(), OBJECT_ID('dbo.SplitDemo'),1, NULL, 'SAMPLED')
WHERE alloc_unit_type_desc = 'IN_ROW_DATA';

After the page split:

Line 1 is located at page 376

Line 2 is located at page 379

Line 3 is located at page 378

Line 4 is located at page 378

Why does it cost so little today?

There is exactly one mechanism through which external fragmentation ever costed something: the read-ahead. During a scan, the engine prefetches pages with large I/Os and an I/O is a contiguous range of the file: one offset, one length, up to 512 KB. When the index is contiguous one I/O brings back 64 pages. When it is not the same pages arrive in several smaller I/Os.

When do these smaller I/Os hurt ? Three conditions must be met at the same time:

• The data is cold: a page already in the buffer pool generates no physical I/O at all
• The operation is a large scan: a seek navigates the B-tree by page ID and does not care about the order
• Each additional I/O has a price

On SSD, NVMe and cloud storage, an additional I/O costs a few dozen microseconds and the physical placement below the file is arbitrary anyway (SSD controllers and SANs put the blocks wherever they want).

External fragmentation survived its own usefulness for another reason: it was a good symptom. It increases because of page splits, and page splits destroy the page density and inflate the transaction log.

Reference

Automatic Index Compaction – SQL Server | Microsoft Learn

Thank you. Amine Haloui

L’article SQL Server: Automatic index compaction in Azure (preview) – Part 1 est apparu en premier sur dbi Blog.

When discussing SQL Server performance, we often hear our performance tool say, ‘An index is missing’, but never ‘There are too many indexes’.

However, in many databases, certain indexes are never used by business queries and they are maintained with every INSERT, UPDATE or DELETE operation.

The question is really simple:

• How much does an unused index actually cost?
• Why is an unused index a problem?

Every time there is a change to the data, SQL Server must also update all the relevant indexes, and that is not free.

As an example, I’m going to use a widely recognised database: the famous DynamicsBC.

The AUDIT

The first step in any study is always to carry out an inventory to gather all the data required for the analysis. I need the size of the database, size of each data file, size of the data and size of the indexes

I will use a query using the DMV: sys.dm_db_partition_stats

USE DynamicsBC;
GO

SELECT
DB_NAME() AS DatabaseName,
CAST(SUM(reserved_page_count) * 8.0 / 1024 AS DECIMAL(18,2)) AS TotalSizeMB,
CAST(SUM(used_page_count) * 8.0 / 1024 AS DECIMAL(18,2)) AS UsedSizeMB,
CAST((SUM(reserved_page_count) - SUM(used_page_count)) * 8.0 / 1024 AS DECIMAL(18,2)) AS FreeSizeMB,
CAST(SUM(
CASE
WHEN index_id < 2 THEN in_row_data_page_count
+ lob_used_page_count
+ row_overflow_used_page_count
ELSE 0
END
) * 8.0 / 1024 AS DECIMAL(18,2)) AS DataSizeMB,
CAST(SUM(
CASE
WHEN index_id >= 2 THEN used_page_count
ELSE 0
END
) * 8.0 / 1024 AS DECIMAL(18,2)) AS IndexSizeMB
FROM sys.dm_db_partition_stats;
GO

If we look at the ratio between the indexes and the data, we get:

64966.91/92898.87 × 100 = ~70 %

A ratio exceeding 50–60 % often requires an analysis of unused indexes, index overlap and compression.

YES, we are good with this database!

Now, let see if we have unused Indexes I use a script that I created years ago for DWH audit:

/******************************************************************************************/
--Summary
/******************************************************************************************/
USE [master];

DECLARE @sqlcmd NVARCHAR(max);
DECLARE @ExcludeFlag BIT = 0;
DECLARE @IncludeDB VARCHAR(1000);
DECLARE @ExcludeDB VARCHAR(1000);


SET  @IncludeDB = 'DynamicsBC'

SELECT @ExcludeFlag = 0;
--SELECT @ExcludeDB = 'distribution,master,model,msdb,tempdb';
 

DECLARE @InDBList TABLE (indb SYSNAME);
DECLARE @ExDBList TABLE (exdb SYSNAME);
DECLARE @dbList TABLE (dbID INT NOT NULL PRIMARY KEY, dbName SYSNAME NOT NULL);


IF (@ExcludeFlag = 0)
BEGIN
	INSERT INTO @InDBList SELECT * FROM string_split(@IncludeDB,',');
	INSERT INTO @dbList (dbID, dbName) 
		SELECT d.database_id, d.name 
		FROM sys.databases d 
		WHERE d.name IN (SELECT indb FROM @InDBList);
END
ELSE
BEGIN
--	INSERT INTO @ExDBList SELECT * FROM string_split(@ExcludeDB,',');
	INSERT INTO @dbList (dbID, dbName) 
		SELECT d.database_id, d.name 
		FROM sys.databases d 
--		WHERE d.name NOT IN (SELECT exdb FROM @ExDBList);
--		WHERE d.name NOT IN ('distribution','master','model','msdb','tempdb');

END

/*********************************************************************
Creation of the result table variable to store the information
--*********************************************************************/

IF (SELECT OBJECT_ID('tempdb.dbo.#dbUnUsedIndex')) IS NOT NULL  
	DROP TABLE dbo.#dbUnUsedIndex
CREATE TABLE #dbUnUsedIndex (
	dbID INT NOT NULL,
	dbName SYSNAME NOT NULL,
	objName SYSNAME NULL,
	idxName SYSNAME NULL,
	idxID INT NULL,
	UserSeek BIGINT NULL,
	UserScans BIGINT NULL,
	UserLookups BIGINT NULL,
	UserUpdates BIGINT NULL,
	nbRows BIGINT NULL,
	dropStatement NVARCHAR(1000) NULL)

DECLARE @dbID INT;
DECLARE @dbName SYSNAME;
DECLARE cur_dblst CURSOR 
  LOCAL STATIC READ_ONLY FORWARD_ONLY
FOR 
SELECT dbID, dbName
FROM @dbList;

OPEN cur_dblst
FETCH NEXT FROM cur_dblst INTO @dbID, @dbName
WHILE @@FETCH_STATUS = 0
BEGIN 
    --Do something with Id here

	SELECT @sqlcmd = N'
		USE [' + @dbName + ']'
		+ ' SELECT
			@dbID
			, @dbName			
			, o.name AS objName
			, i.name AS idxName
			, i.index_id AS idxID
			, dm_ius.user_seeks AS UserSeek
			, dm_ius.user_scans AS UserScans
			, dm_ius.user_lookups AS UserLookups
			, dm_ius.user_updates AS UserUpdates
			, p.TableRows AS nbRows
			, ''DROP INDEX '' + QUOTENAME(i.name)
			+ '' ON '' + QUOTENAME(s.name) + ''.''
			+ QUOTENAME(OBJECT_NAME(dm_ius.OBJECT_ID)) AS ''dropStatement''
		FROM sys.dm_db_index_usage_stats dm_ius
			INNER JOIN sys.indexes i ON i.index_id = dm_ius.index_id AND dm_ius.OBJECT_ID = i.OBJECT_ID
			INNER JOIN sys.objects o ON dm_ius.OBJECT_ID = o.OBJECT_ID
			INNER JOIN sys.schemas s ON o.schema_id = s.schema_id
			INNER JOIN (SELECT SUM(p.rows) TableRows, p.index_id, p.OBJECT_ID
			FROM sys.partitions p GROUP BY p.index_id, p.OBJECT_ID) p
			ON p.index_id = dm_ius.index_id AND dm_ius.OBJECT_ID = p.OBJECT_ID
		WHERE OBJECTPROPERTY(dm_ius.OBJECT_ID,''IsUserTable'') = 1
			AND dm_ius.database_id = DB_ID()
			AND i.type_desc = ''nonclustered''
			AND i.is_primary_key = 0
			AND i.is_unique_constraint = 0
		ORDER BY (dm_ius.user_seeks + dm_ius.user_scans + dm_ius.user_lookups) ASC; '

	INSERT INTO #dbUnUsedIndex (
					dbID,
					dbName,
					objName,
					idxName,
					idxID,
					UserSeek,
					UserScans,
					UserLookups,
					UserUpdates,
					nbRows,
					dropStatement)
	EXEC sp_executesql  @sqlCmd, N'@dbID INT, @dbName SYSNAME', @dbID, @dbName

    FETCH NEXT FROM cur_dblst INTO @dbID, @dbName
END
CLOSE cur_dblst
DEALLOCATE cur_dblst

SELECT Count(*) FROM #dbUnUsedIndex

SELECT * FROM #dbUnUsedIndex
ORDER BY dbName, objName


 IndexInfo;
GO

For the total number of Indexes, we use this simple query:

SELECT COUNT(*) AS TotalIndexes
FROM dynamicsBC.sys.indexes i
INNER JOIN sys.tables t
ON i.object_id = t.object_id
WHERE i.index_id > 0;

We have 998 unused Index for a total of 19494 Index. This is 5% of the Indexes.
In this case, we will not win a lot. 5% is good but we can have a look because we never known…

The IMPACT

The first impact is on DML INSERT and UPDATE operations.

INSERT

With every insertion, the data is written to the table and several index structures must also be updated, even when one of these indexes is never used by a query.

So we have the following sequence:

INSERT –> write to the table –> update index 1  –> update index 2  –> update index 3  –> update index 4 ….

This is even more costly for this command because, for each operation, the sequence is as follows:

UPDATE

UPDATE –> Delete the old index entry + Create the new entry for each index

The result of all of this is more I/O, more CPU usage and also more entries in the Transaction Log

On heavily used transactional tables, a few unused indexes can account for several per cent of additional load.

The second impact is the storage and the backup

If the unused index represents ~100B of the database.

If 100 GB of indexes are never used, that’s 100 GB backed up unnecessarily, 100 GB restored unnecessarily and, above all, 100 GB stored unnecessarily and just for one environment.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s near 1 TB!

The third impact is on the maintenance plan

The following operations must also handle unused indexes:
– Rebuild Index
– Reorganise Index
– CheckDB
– Backups
– Restores

Each additional index extends the maintenance windows…

The ADVISE

You really need to be careful before deleting anything, as an unused index is not necessarily useless.

You should check several factors before you delete unused indexes:
– Full business cycle
– End/Begin of month
– Annual processing
– Reporting
– ETL
– SQL Agent jobs

I generally recommend observing the index between 1 and 3 months before deleting it and without restart. A restart will reset the DMV used for the stats… And always keep the script for recreating the index to hand!

The Green SQL Server DBA Score

Just for fun, we’re going to set up a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

An unused index is a bit like car that never gets driven:
it takes up space;it costs money;it requires maintenance;but it produces no value.

The Green SQL Server DBA doesn’t just seek to add indexes.
They aim to retain only those that deliver measurable business value.

See you soon for the next one!

L’article Green SQL Server DBA Tips  #1 – Are unnecessary indexes cluttering up your database? est apparu en premier sur dbi Blog.

A production instance, mid-afternoon, nothing unusual on any dashboard. An engineer opens a transaction to patch a single row while investigating a data issue:

BEGIN TRANSACTION;
UPDATE dbo.Orders SET Status = 'Reviewed' WHERE OrderId = 482193;

No COMMIT. No ROLLBACK. The tab gets buried under three others, the investigation moves on, and the lock is still held an hour later.

Every query, every batch, every login needs a worker thread to execute on. That pool is not infinite, it is sized by max worker threads, either left on its computed default or pinned to a fixed number.

SELECT name, value_in_use FROM sys.configurations WHERE name = 'max worker threads';

name                                value_in_use
----------------------------------- -------------------------------------------------------------------------------------------------------------------------
max worker threads                  128

What does “Sleeping” really mean?

SQL Server schedules work cooperatively, not preemptively. Each worker is handed a quantum (4 milliseconds) to run before it is expected to voluntarily yield the scheduler to the next runnable task. This is the mechanism behind SOS_SCHEDULER_YIELD: a worker that still has work to do, but whose quantum has expired, stepping aside so someone else gets a turn.

None of this applies to the open transaction from earlier. A session that has issued no command has no task and holds no worker. Its status in sys.dm_exec_sessions is sleeping, not running, not suspended.

SELECT
    s.session_id,
    s.status AS session_status,
    ct.text
FROM sys.dm_exec_sessions s
LEFT JOIN sys.dm_exec_requests r ON s.session_id = r.session_id
LEFT JOIN sys.dm_exec_connections c ON s.session_id = c.session_id
OUTER APPLY sys.dm_exec_sql_text(c.most_recent_sql_handle) ct
WHERE s.session_id = 54;

session_id session_status                 text
---------- ------------------------------ --------------------------------------------------------------------------------------------------------------------
54         sleeping                       UPDATE dbo.Orders SET Status = 'Reviewed' WHERE OrderId = 482193;

It is not waiting for a quantum, because it is not competing for one. The lock it holds costs the engine nothing in scheduling terms; it is bookkeeping in the lock manager, entirely separate from the worker pool.

Two hundred sessions walk into a lock

Let’s say that the application wants to confirm that the orders has been reviewed now it’s in the processed state.

BEGIN TRANSACTION;
UPDATE dbo.Orders SET Status = 'Processed' WHERE OrderId = 482193;

Seeing that the query didn’t complete to update the item, it will keep sending this transaction again and again, sending it 200 times let’s say.

Unlike the sleeping session above, each of these has issued a command. Each one is granted a worker to execute it, immediately hits the lock, and transitions to suspended, waiting on LCK_M_X.

wait_type                   waiting_tasks_count       wait_time_ms
THREADPOOL                      521                      2881770
SOS_SCHEDULER_YIELD             710                      41

 session_id     status        wait_type        wait_time   blocking_session_id 
    68         suspended       LCK_M_X          29873             54
   ...
   206         suspended       LCK_M_X          29478             68
   207         suspended       LCK_M_X          29478             68

scheduler_id runnable_tasks_count work_queue_count active_workers_count
       0              0               19                 43
       1              0               5                  45
       2              0               10                 45
       3              0               2                  44

active_workers    max_workers_count       
         205                    128

Note: max_workers_count only counts the user-facing pool; internal system threads, including the DAC’s own reserved worker used to capture this very output, sit outside that ceiling.

The worker is not released while the task waits. It stays attached to the suspended task for the entire duration of the block, doing nothing, simply reserved, waiting for the resource (the order line to update) to be available for updates.

The remainder cannot even be granted a worker to start waiting. They queue behind everyone else, and eventually give up entirely:

Login timeout expired
Login/Query timeout: 15/0 seconds

By this point the server has simply stopped accepting new connections.

When the fire exit is also on fire

Releasing the original lock should be the easy part: switch back to the session from the very first transaction, issue a ROLLBACK, and watch everything clear. Except that session, which has been sitting sleeping and worker-free this whole time, now has to issue a command of its own. And issuing a command means asking the pool for a worker (the same exhausted pool every other session is already queued for). The session responsible for the deadlock has no priority for fixing it. It gets in line like everyone else, behind two hundred sessions it created the conditions for.

When DAC is the last resort

This is where the Dedicated Admin Connection comes in the game. It runs on its own scheduler, with a worker reserved outside the regular pool, built specifically for an instance too exhausted to serve itself.

sqlcmd -A -S"." -E

SELECT blocking_session_id
FROM sys.dm_exec_requests
WHERE blocking_session_id <> 0;

KILL 54;

Note: the “.” here resolves to the local default instance but unlike an ordinary local connection (which typically uses Shared Memory), the DAC always connects over its own dedicated TCP listener on the loopback adapter, regardless of protocol settings on the port 1434 or a dynamic one (full documentation here).

The KILL forces the rollback from outside the exhausted pool entirely. Workers free up in cascade, and the two hundred suspended sessions complete their updates and release their own.

Final thoughts

In this example, we set the parameter max worker threads to 128 to easily saturate the worker threads. However, the default value for max worker threads is 0, which lets SQL Server compute the number of worker threads automatically at startup based on the number of logical CPUs and the platform architecture. Microsoft best practice can be found here and shows the following table:

And the key take-away from this experiment:

• Sleeping costs nothing, suspended costs a worker. The distinction between an idle transaction and a blocked one is the entire mechanism behind this incident: both hold a lock, only one of them holds a thread.
• The scheduler’s quantum explains CPU pressure, not threadpool exhaustion. Yielding after 4ms is about sharing a CPU among runnable workers; it has nothing to do with how many workers exist in the first place.
• The session that caused the block is not exempt from the consequences of the block. It has to compete for a worker like anything else, the moment it tries to clean up after itself.
• Never let a statement end without a COMMIT or a ROLLBACK.

L’article When an idle transaction starves the worker pool (THREADPOOL) est apparu en premier sur dbi Blog.

Monitoring Microsoft SQL Server database files is an important part of infrastructure supervision.
A database reaching its storage limit can rapidly lead to application outages, transaction failures, or even complete service interruptions.

The Initial Problem

In SQL Server, retrieving file information is relatively simple when working directly inside SQL Server Management Studio

• database file size,
• used space,
• free space,
• auto-growth configuration,
• or maximum file size

A common query used by many DBAs is the following:

USE [TEST]

GO

SELECT
  DB_NAME(database_id) AS DBName,
  Name AS Logical_Name,
  Physical_Name AS [PhysicalName],
  CAST(ROUND(((size)/128/1024), 2) AS float) AS [SizeGB],
  CAST(ROUND((CAST(FILEPROPERTY(name, 'SpaceUsed') AS INT)/128/1024), 2) AS float) AS [SpaceUsedGB],
  CAST(ROUND((size/128/1024 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS INT)/128/1024), 2) AS float) AS [FreeSpaceGB],
  ROUND(ISNULL(((CAST((fileproperty(name, 'SpaceUsed'))/128/1024 as float)) / NULLIF((CAST(size/128/1024 as float)), 0)), 0)*100, 2) as [SpaceUsed%],
  CASE WHEN [max_size] = -1 THEN [max_size] ELSE CAST(ROUND(((max_size)/128/1024), 2) AS float) END AS [MaxSizeGB],
  CAST(ROUND((max_size/128/1024 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS INT)/128/1024), 2) AS float) AS [FreeSpaceGBtoMaxSize],
  ROUND(ISNULL(((CAST((fileproperty(name, 'SpaceUsed'))/128/1024 as float)) / NULLIF((CAST(max_size/128/1024 as float)), 0)), 0)*100, 2) as [SpaceUsed%toMaxSize]
FROM sys.master_files
WHERE DB_NAME(database_id) = 'TEST'
ORDER BY Logical_Name

The initial query works perfectly for manual analysis in SQL Server Management Studio.
However, integrating it into Zabbix becomes more complicated because sys.database_files only returns information from the current database context.

This means the query must be executed individually for every database using:

USE [DatabaseName]

In small environments this remains manageable, but in larger infrastructures containing dozens or even hundreds of databases, maintaining static queries quickly becomes impractical.

To automate this process, we use:

sp_MSforeachdb

This SQL Server procedure dynamically executes the same query across every database on the instance.

Example:

EXEC sp_MSforeachdb '
USE [?]

SELECT
    DB_NAME(),
    name,
    physical_name
FROM sys.database_files
'

The ? placeholder is automatically replaced by the database name during execution, allowing SQL Server to iterate through all databases dynamically.

At first glance, this seems to solve the problem entirely.
However, another limitation quickly appears: sp_MSforeachdb returns one independent result set per database.

Instead of producing a single structured dataset, SQL Server generates multiple separate tables:

Database 1 result
-----------------
file1
file2

Database 2 result
-----------------
file1
file2

While this output remains perfectly readable for a DBA inside SQL Server Management Studio, it becomes difficult to exploit inside Zabbix, especially for Low-Level Discovery and dependent items.

To solve this limitation, we redesigned the query architecture to centralize all results into a single temporary table, allowing Zabbix to consume one normalized dataset fully compatible with automatic discovery and scalable monitoring.

Why Zabbix Cannot Easily Use This Output

Zabbix works much better when:

• the result is normalized,
• the structure is predictable,
• and all rows belong to a single dataset.

With multiple independent result sets:

• Low-Level Discovery becomes difficult,
• JSON conversion becomes complicated,
• dependent items cannot parse values correctly,
• preprocessing becomes unreliable.

The issue is therefore no longer the database context itself. The issue becomes the query output structure.

Building a Zabbix-Compatible Query

To make the output usable inside Zabbix, the query had to be redesigned completely.

The objective was no longer simply to retrieve MSSQL file information, but to transform multiple independent database results into a single normalized dataset compatible with Low-Level Discovery.

The new query therefore introduces three important concepts:

• dynamic iteration through all databases using sp_MSforeachdb,
• centralized data collection using a global temporary table,
• and a final unified output consumable by Zabbix dependent items.

The core logic becomes inside the sp_MSforeachdb execution.

INSERT INTO ##Results

Each database inserts its rows into the same centralized structure instead of returning independent result sets.

At the end of the execution, the query simply returns:

SELECT * FROM ##Results

The final output is now normalized instead of multiple disconnected tables.

The complete query

SET NOCOUNT ON;

DECLARE @Granularity VARCHAR(10) = NULL
DECLARE @Database_Name sysname = NULL
 
DECLARE @SQL VARCHAR(5000)

IF EXISTS (SELECT NAME FROM tempdb..sysobjects WHERE NAME = '##Results')
BEGIN
    DROP TABLE ##Results
END

CREATE TABLE ##Results (
    [Database Name] sysname,
    [File Name] sysname,
    [Physical Name] NVARCHAR(260),
    [File Type] VARCHAR(4),
    [Total Size in Bytes] BIGINT,
    [Available Space in Bytes] BIGINT,
    [Growth Units] VARCHAR(20),
    [Max File Size in Bytes] BIGINT
)

SELECT @SQL =
'USE [?] 
INSERT INTO ##Results(
    [Database Name],
    [File Name],
    [Physical Name],
    [File Type],
    [Total Size in Bytes],
    [Available Space in Bytes],
    [Growth Units],
    [Max File Size in Bytes]
)
SELECT 
    DB_NAME(),
    [name],
    physical_name,
    CASE type
        WHEN 0 THEN ''Data''
        WHEN 1 THEN ''Log''
    END,

    -- TOTAL SIZE (bytes) SAFE
    CAST(size AS BIGINT) * 8 * 1024,

    -- FREE SPACE (bytes) SAFE
    (CAST(size AS BIGINT) - CAST(FILEPROPERTY([name], ''SpaceUsed'') AS BIGINT)) * 8 * 1024,

    -- GROWTH
    CASE is_percent_growth
        WHEN 1 THEN CAST(growth AS varchar(20)) + ''%''
        ELSE CAST(CAST(growth AS BIGINT) * 8 * 1024 AS varchar(20))
    END,

    -- MAX SIZE (bytes) SAFE
    CASE max_size
        WHEN -1 THEN NULL
        WHEN 268435456 THEN NULL
        ELSE CAST(max_size AS BIGINT) * 8 * 1024
    END

FROM sys.database_files
ORDER BY type, file_id
'

EXEC sp_MSforeachdb @SQL

-- =========================
-- RESULTATS
-- =========================

IF @Database_Name IS NULL
BEGIN
    IF @Granularity = 'Database'
    BEGIN
        SELECT
            T.[Database Name],

            T.[Total Size in Bytes] AS [DB Size (Bytes)],
            T.[Available Space in Bytes] AS [DB Free (Bytes)],
            T.[Consumed Space in Bytes] AS [DB Used (Bytes)],

            D.[Total Size in Bytes] AS [Data Size (Bytes)],
            D.[Available Space in Bytes] AS [Data Free (Bytes)],
            D.[Consumed Space in Bytes] AS [Data Used (Bytes)],
            CEILING(CAST(D.[Available Space in Bytes] AS decimal(20,2)) / NULLIF(D.[Total Size in Bytes],0) * 100) AS [Data Free %],

            L.[Total Size in Bytes] AS [Log Size (Bytes)],
            L.[Available Space in Bytes] AS [Log Free (Bytes)],
            L.[Consumed Space in Bytes] AS [Log Used (Bytes)],
            CEILING(CAST(L.[Available Space in Bytes] AS decimal(20,2)) / NULLIF(L.[Total Size in Bytes],0) * 100) AS [Log Free %]

        FROM
        (
            SELECT [Database Name],
                SUM([Total Size in Bytes]) AS [Total Size in Bytes],
                SUM([Available Space in Bytes]) AS [Available Space in Bytes],
                SUM([Total Size in Bytes] - [Available Space in Bytes]) AS [Consumed Space in Bytes]
            FROM ##Results
            GROUP BY [Database Name]
        ) T

        INNER JOIN
        (
            SELECT [Database Name],
                SUM([Total Size in Bytes]) AS [Total Size in Bytes],
                SUM([Available Space in Bytes]) AS [Available Space in Bytes],
                SUM([Total Size in Bytes] - [Available Space in Bytes]) AS [Consumed Space in Bytes]
            FROM ##Results
            WHERE [File Type] = 'Data'
            GROUP BY [Database Name]
        ) D ON T.[Database Name] = D.[Database Name]

        INNER JOIN
        (
            SELECT [Database Name],
                SUM([Total Size in Bytes]) AS [Total Size in Bytes],
                SUM([Available Space in Bytes]) AS [Available Space in Bytes],
                SUM([Total Size in Bytes] - [Available Space in Bytes]) AS [Consumed Space in Bytes]
            FROM ##Results
            WHERE [File Type] = 'Log'
            GROUP BY [Database Name]
        ) L ON T.[Database Name] = L.[Database Name]

        ORDER BY T.[Database Name]
    END
    ELSE
    BEGIN
        SELECT
            [Database Name],
            [File Name],
            [Physical Name],
            [File Type],
            [Total Size in Bytes] AS [DB Size (Bytes)],
            [Available Space in Bytes] AS [DB Free (Bytes)],
            CEILING(CAST([Available Space in Bytes] AS decimal(20,2)) / NULLIF([Total Size in Bytes],0) * 100) AS [Free Space %],
            [Growth Units],
            [Max File Size in Bytes] AS [Grow Max Size (Bytes)]
        FROM ##Results
    END
END
ELSE
BEGIN
    IF @Granularity = 'Database'
    BEGIN
        SELECT
            T.[Database Name],

            T.[Total Size in Bytes] AS [DB Size (Bytes)],
            T.[Available Space in Bytes] AS [DB Free (Bytes)],
            T.[Consumed Space in Bytes] AS [DB Used (Bytes)],

            D.[Total Size in Bytes] AS [Data Size (Bytes)],
            D.[Available Space in Bytes] AS [Data Free (Bytes)],
            D.[Consumed Space in Bytes] AS [Data Used (Bytes)],
            CEILING(CAST(D.[Available Space in Bytes] AS decimal(20,2)) / NULLIF(D.[Total Size in Bytes],0) * 100) AS [Data Free %],

            L.[Total Size in Bytes] AS [Log Size (Bytes)],
            L.[Available Space in Bytes] AS [Log Free (Bytes)],
            L.[Consumed Space in Bytes] AS [Log Used (Bytes)],
            CEILING(CAST(L.[Available Space in Bytes] AS decimal(20,2)) / NULLIF(L.[Total Size in Bytes],0) * 100) AS [Log Free %]

        FROM
        (
            SELECT [Database Name],
                SUM([Total Size in Bytes]) AS [Total Size in Bytes],
                SUM([Available Space in Bytes]) AS [Available Space in Bytes],
                SUM([Total Size in Bytes] - [Available Space in Bytes]) AS [Consumed Space in Bytes]
            FROM ##Results
            WHERE [Database Name] = @Database_Name
            GROUP BY [Database Name]
        ) T

        INNER JOIN
        (
            SELECT [Database Name],
                SUM([Total Size in Bytes]) AS [Total Size in Bytes],
                SUM([Available Space in Bytes]) AS [Available Space in Bytes],
                SUM([Total Size in Bytes] - [Available Space in Bytes]) AS [Consumed Space in Bytes]
            FROM ##Results
            WHERE [File Type] = 'Data'
              AND [Database Name] = @Database_Name
            GROUP BY [Database Name]
        ) D ON T.[Database Name] = D.[Database Name]

        INNER JOIN
        (
            SELECT [Database Name],
                SUM([Total Size in Bytes]) AS [Total Size in Bytes],
                SUM([Available Space in Bytes]) AS [Available Space in Bytes],
                SUM([Total Size in Bytes] - [Available Space in Bytes]) AS [Consumed Space in Bytes]
            FROM ##Results
            WHERE [File Type] = 'Log'
              AND [Database Name] = @Database_Name
            GROUP BY [Database Name]
        ) L ON T.[Database Name] = L.[Database Name]

        ORDER BY T.[Database Name]
    END
    ELSE
    BEGIN
        SELECT
            [Database Name],
            [File Name],
            [Physical Name],
            [File Type],
            [Total Size in Bytes] AS [DB Size (Bytes)],
            [Available Space in Bytes] AS [DB Free (Bytes)],
            CEILING(CAST([Available Space in Bytes] AS decimal(20,2)) / NULLIF([Total Size in Bytes],0) * 100) AS [Free Space %],
            [Growth Units],
            [Max File Size in Bytes] AS [Grow Max Size (Bytes)]
        FROM ##Results
        WHERE [Database Name] = @Database_Name
    END
END

DROP TABLE ##Results

Discovery in Zabbix

Once the query has been implemented in Zabbix, the next step is to create a Low-Level Discovery (LLD) rule in order to automatically discover databases and files, then generate the associated dependent items and triggers dynamically.

A dedicated blog explaining how to configure Low-Level Discovery in Zabbix is available here: Create DISCO Put the link here

Conclusion

Monitoring MSSQL database files in Zabbix can become challenging when working with multiple databases and dynamic environments.
While the initial query is perfectly suitable for manual analysis, its structure and dependency on the current database context make it difficult to integrate directly into Zabbix.

By using sp_MSforeachdb together with a centralized temporary table, we can transform multiple independent result sets into a single normalized dataset fully compatible with Zabbix Low-Level Discovery and dependent items.

This approach provides a scalable and reusable solution capable of automatically monitoring database and log file growth across an entire SQL Server instance while significantly reducing manual configuration and maintenance efforts.

L’article Monitoring MSSQL Database Files in Zabbix est apparu en premier sur dbi Blog.