This is something we often see among our customers. If we assume that we carry out a migration project roughly every five years, this means we’re keeping databases that are taking up space for no reason.

We, however, are Green SQL Server DBA; we’ll look to manage it correctly.

The AUDIT

First of all, what do we need to analyse?

This does not apply to all databases, but only to those that are obsolete.

They can be categorised into two types:

• Databases Never Accessed
  • No application connections detected
  • No recent transactions or queries after a period (3 months for exemple)
• Abandoned Databases
  • Linked to cancelled projects
  • No activity and nobody tell it

To see databases never accessed and abandoned databases I use the DMV sys.dm_db_index_usage_stats to see if I have an access to the database (read or write):

SELECT d.name AS DatabaseName, MAX(ius.last_user_seek) AS LastSeek,MAX(ius.last_user_scan) AS LastScan,
    MAX(ius.last_user_lookup) AS LastLookup,

    (
        SELECT MAX(v.ActivityDate)
        FROM (VALUES
                (MAX(ius.last_user_seek)),
                (MAX(ius.last_user_scan)),
                (MAX(ius.last_user_lookup))
             ) v(ActivityDate)
    ) AS LastReadActivity,

    MAX(ius.last_user_update) AS LastWriteActivity,

    DATEDIFF(DAY,
        (
            SELECT MAX(v.ActivityDate)
            FROM (VALUES
                    (MAX(ius.last_user_seek)),
                    (MAX(ius.last_user_scan)),
                    (MAX(ius.last_user_lookup))
                 ) v(ActivityDate)
        ),
        GETDATE()
    ) AS DaysSinceLastRead,

    DATEDIFF(DAY,
        MAX(ius.last_user_update),
        GETDATE()
    ) AS DaysSinceLastWrite

FROM sys.databases d
LEFT JOIN sys.dm_db_index_usage_stats ius
    ON d.database_id = ius.database_id
WHERE d.database_id > 4  -- Exclude system databases
GROUP BY d.name
ORDER BY LastWriteActivity ASC;

We can see on the first databases in the list, that we have null everywhere… This means that the databases were not solicited since the last restart… Good candidate for unused databases!

This brings us to the next point.

The ANALISYS

If we can see that there is little activity on the database, we can, of course, carry out an initial analysis using criteria such as these:

< 30 days –> the database is still active

30–180 days –> the database is on hold pending to be dropped

> 180 days –> the database is a strong candidate to be dropped

Of course, it also depends on your environment – whether it’s dev, test or prod – and the criteria will vary accordingly.

The IMPACT

Like in my precedent post, we will have same impacts but not for queries of courses!

The first impact is the storage and the backup

If the database represents ~100 GB on the disk , it’s also 100 GB backed up unnecessarily.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 1 TB!

The third impact is on the maintenance plan

The following operations must also handle the unused database:

– CheckDB
– Backups

As the database is not used, the index rebuild/reorg & update Stats will be fast!

Each unused database extends the maintenance windows for the checkdb and backups…

The ADVISE

The first advice would be to plan well in advance.

Indeed, when creating or restoring a database, it is important to know its lifecycle.

Don’t hesitate to challenge the application owners on a deadline and make sure to add it to your calendar +1day.

The second advice is to set up a monthly monitoring process prior to Windows or SQL Server patch cycles so that you can identify these unnecessary databases.

And before dropping a database, I advice you to put the database offline and see if after some days somebody complaints…

In the majority of cases, you will not have reactions, believe me!

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will let the unused databases until the next migration.

A Green SQL Server DBA will check it periodically and manage it.

An unused database is like a car that never leaves the garage.
 It still costs money, requires maintenance, and consumes resources.
The Green SQL Server DBA notices it before it becomes waste…

Think about it and begin now to see to monitor your unused databases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #3 – The hidden cost of forgotten databases est apparu en premier sur dbi Blog.

Can this also be applied to PostgreSQL?

In short, yes, it is possible. Let’s see how.

Here is the list of the previous blog posts:

• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-2-3/
• https://www.dbi-services.com/blog/sql-server-snapshot-backup-and-restore-with-proxmox-zfs-rest-api-with-sql-server-2025-3-3/

Partitioning and filesystem

We will reuse the sqlpool ZFS pool created in the first part of this series.

We identify the 300 GB disk attached to the VM. In our case, it is /dev/sdb, backed by the sqlpool/pve/vm-307-disk-0 zvol on the Proxmox side:

lsblk

We create a single partition of type Linux filesystem:

sudo sgdisk -n 1:0:0 -t 1:8300 /dev/sdb

We format the partition with XFS, which is the most commonly recommended filesystem for PostgreSQL data directories:

sudo mkfs.xfs -L pgdata /dev/sdb1 -f

We verify the result:

ahi@pgl:~$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="pgdata" UUID="028afa2f-7bb3-4a40-92aa-91c1a33f18ae9" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="bd54f285-092e-4b1b-ba5e-6877f054fa7"

Mountpoint:

We create the mount point:

sudo mkdir -p /pgdata

Persistent mount via fstab:

We add the mount entry to /etc/fstab using the filesystem label rather than the device name. The device name (/dev/sdb) may change if disks are added or removed while the label remains stable:

echo 'LABEL=pgdata /pgdata xfs noatime,nodiratime 0 2' | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /pgdata

We verify that the volume is mounted:

df -h /pgdata

PostgreSQL installation

We install PostgreSQL 18 from the official PGDG repository, which provides the latest PostgreSQL versions for Ubuntu:

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
sudo apt install -y postgresql-18

Cluster creation on /pgdata:

The Ubuntu packages create a default cluster under /var/lib/postgresql. This is not what we want. The data files and the WAL must both reside on the ZFS-backed volume, so that a single ZFS snapshot captures a consistent state of the database. If they were on different volumes, the snapshot would not be atomic.

We drop the default cluster and recreate it on /pgdata:

ahi@pgl:~$ sudo pg_dropcluster --stop 18 main
sudo install -d -o postgres -g postgres -m 700 /pgdata/18
sudo pg_createcluster -d /pgdata/18/main 18 main
sudo systemctl enable --now postgresql@18-main
Creating new PostgreSQL cluster 18/main ...
/usr/lib/postgresql/18/bin/initdb -D /pgdata/18/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /pgdata/18/main ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
             Ver Cluster Port Status Owner     Data directory  Log file
             18  main    5432 down   postgres  /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql@18-main.service → /usr/lib/systemd/system/postgresql@.service.

We verify that the cluster is online and located on the right volume:

ahi@pgl:~$ pg_lsclusters
Ver Cluster Port Status Owner     Data directory  Log file
18  main    5432 online <unknown> /pgdata/18/main /var/log/postgresql/postgresql-18-main.log
ahi@pgl:~$ sudo -u postgres psql -c "SHOW data_directory;"
 data_directory
-----------------
 /pgdata/18/main
(1 row)

ahi@pgl:~$ sudo -u postgres psql -c "SELECT version();"
                                                             version
-------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
(1 row)

We can also confirm that the WAL directory lives inside the data directory, and therefore on the zvol:

ls -ld /pgdata/18/main/pg_wal

Creating a large database

We need a database large enough to make traditional backup and restore operations time-consuming. In the SQL Server part of this series, we used the StackOverflow database (about 207 GB). For PostgreSQL, we use pgbench, the benchmarking tool shipped with PostgreSQL.

We create the database and initialize it with a scale factor of 10000. This produces a database of approximately 146 GB, with 1 billion rows in the pgbench_accounts table:

sudo -u postgres createdb bench
sudo -u postgres pgbench -i -s 10000 --partitions=8 bench

A few minutes later:

We can monitor the data growth during the initialization:

watch -n 30 'df -h /pgdata'

A few minutes later:

On the Proxmox side:

After some time, the process completes:

vacuuming...

creating primary keys...
done in 1559.55 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 598.12 s, vacuum 675.53 s, primary keys 285.88 s).
ahi@pgl:~$

We check the database size:

We run a checkpoint before taking the snapshot. The recovery process starts replaying the WAL from the last checkpoint. By running it right before the snapshot, almost nothing needs to be replayed when the database starts after a restore:

sudo -u postgres psql -c "CHECKPOINT;"

Comparison with SQL Server:

On the SQL Server side, we had to run SUSPEND_FOR_SNAPSHOT_BACKUP and BACKUP WITH METADATA_ONLY. On the PostgreSQL side, none of that is needed.

The data files and the WAL are on the same zvol. An atomic ZFS snapshot therefore captures a state equivalent to a power loss, and PostgreSQL is designed to recover cleanly from that state through crash recovery: the WAL is replayed from the last checkpoint. This is documented and officially supported.

The snapshot is the backup. There is no .bkm file, no metadata backup.

Snapshot process flow

On the Proxmox side, we create the snapshot and protect it with a hold:

SNAP="sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)" 
zfs snapshot "$SNAP" 
zfs hold sqlsnap "$SNAP" 
echo "$SNAP"

The hold protects the snapshot from an accidental destruction, as we did in part 2 of this series. We note the exact snapshot name, it will be needed for the restore.

The database stays online during the whole operation. No I/O freeze is required.

We list the snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We drop the database then we restore the snapshot:

We run the snapshot restore procedure. On the VM, we stop the cluster and unmount the volume:

sudo systemctl stop postgresql@18-main
sudo umount /pgdata

On the Proxmox side, we want to restore our snapshot. We can list the available snapshots:

zfs list -t snapshot -r sqlpool/pve/vm-307-disk-0

We roll back the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T165409

On the VM, we mount the volume and start the service:

sudo mount /pgdata
sudo systemctl start postgresql@18-main

We check a few elements in the logs:

sudo tail -30 /var/log/postgresql/postgresql-18-main.log

The service shutdown, then the restart:

UTC [232108] LOG:  database system is shut down
UTC [233968] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
UTC [233968] LOG:  listening on IPv4 address "0.0.0.0", port 5432
UTC [233968] LOG:  listening on IPv6 address "::", port 5432
UTC [233968] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

PostgreSQL detects that the database was not shut down properly and replays the WAL. This is the same crash recovery mechanism as in SQL Server. Finally, the database starts.

UTC [233974] LOG:  database system was not properly shut down; automatic recovery in progress
UTC [233974] LOG:  redo starts at 20/2CB76278
UTC [233974] LOG:  invalid record length at 20/2CB76380: expected at least 24, got 0
UTC [233974] LOG:  redo done at 20/2CB76348 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
UTC [233974] LOG:  checkpoint starting: end-of-recovery immediate wait
UTC [233972] LOG:  checkpoint complete: wrote 0 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.009 s, total=0.030 s; sync files=0, longest=0.000 s, average=0.005 s; distance=0 kB, estimate=0 kB; lsn=20/2CB76380, redo lsn=20/2CB76380
UTC [233968] LOG:  database system is ready to accept connections

We then verify that the database is available again:

sudo -u postgres psql -d bench -c "SELECT pg_size_pretty(pg_database_size('bench'));"

Consistency proof under load

The previous test was done on a quiesced database: we ran a CHECKPOINT right before the snapshot, and nothing was writing. The real question is: what happens if the snapshot is taken while the database is being written to?

This is where PostgreSQL differs the most from SQL Server. There is no SUSPEND_FOR_SNAPSHOT_BACKUP. We take the snapshot in the middle of the write activity and we let the WAL replay do the work.

We start the load. The built-in pgbench script runs a TPC-B-like transaction: three UPDATE statements on the accounts, tellers and branches tables and one INSERT into the history table:

sudo -u postgres pgbench -c 8 -j 4 -T 300 bench &

While the load is running, we take a snapshot on the Proxmox side:

zfs snapshot sqlpool/pve/vm-307-disk-0@pg_bench_$(date +%Y%m%dT%H%M%S)

zfs hold sqlsnap sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551

No CHECKPOINT this time, no freeze. The database is actively writing while the snapshot is taken.

The state after some time under load:

We stop the service:

sudo systemctl stop postgresql@18-main 
sudo umount /pgdata

We restore the snapshot:

zfs rollback -r sqlpool/pve/vm-307-disk-0@pg_bench_20260803T224551 

We mount the volume, start the service and check the logs:

sudo mount /pgdata 
sudo systemctl start postgresql@18-main

This time the log shows a real recovery:

2026-08-03 20:49:03.244 UTC [234405] LOG:  database system is shut down
2026-08-03 20:50:34.333 UTC [234620] LOG:  starting PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2026-08-03 20:50:34.333 UTC [234620] LOG:  listening on IPv6 address "::", port 5432
2026-08-03 20:50:34.335 UTC [234620] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2026-08-03 20:50:34.343 UTC [234626] LOG:  database system was interrupted; last known up at 2026-08-03 20:44:36 UTC
2026-08-03 20:50:34.381 UTC [234626] LOG:  database system was not properly shut down; automatic recovery in progress
2026-08-03 20:50:34.384 UTC [234626] LOG:  redo starts at 20/7A3618B0
2026-08-03 20:50:38.024 UTC [234626] LOG:  invalid record length at 20/8BC0A4F8: expected at least 24, got 0
2026-08-03 20:50:38.024 UTC [234626] LOG:  redo done at 20/8BC0A4D0 system usage: CPU: user: 0.71 s, system: 0.63 s, elapsed: 3.63 s
2026-08-03 20:50:38.028 UTC [234624] LOG:  checkpoint starting: end-of-recovery immediate wait
2026-08-03 20:50:53.936 UTC [234624] LOG:  checkpoint complete: wrote 105355 buffers (53.6%), wrote 5 SLRU buffers; 0 WAL file(s) added, 17 removed, 0 recycled; write=3.648 s, sync=12.232 s, total=15.911 s; sync files=189, longest=12.223 s, average=0.065 s; distance=287395 kB, estimate=287395 kB; lsn=20/8BC0A4F8, redo lsn=20/8BC0A4F8
2026-08-03 20:50:53.948 UTC [234620] LOG:  database system is ready to accept connections

Three differences compared to the first test:

• The “last known up at” timestamp (20:44:36) does not match a checkpoint we ran manually. It matches the last automatic checkpoint triggered during the load.

• The redo is not instantaneous anymore: 3.63 seconds, replaying about 280 MB of WAL (from LSN 20/7A3618B0 to 20/8BC0A4D0). All the write activity between the last checkpoint and the snapshot had to be replayed. The transactions committed before the snapshot are recovered, the ones that were in flight are rolled back.

• The end-of-recovery checkpoint then writes everything the redo rebuilt in memory: 105355 buffers, 53.6% of the buffer pool. The database is ready to accept connections about 19 seconds after the service start.

The crash recovery completed correctly and the database has been restored. We verify the TPC-B invariant. Each pgbench transaction applies the same delta to the accounts, tellers and branches tables in a single transaction. On a consistent database, the three sums must be equal:

Major drawbacks

• The snapshot covers the whole zvol. All the databases of the cluster are captured and restored together. There is no per-database restore, unlike the METADATA_ONLY approach on SQL Server which targets a single database.
• There is no backup history. SQL Server records the metadata backup in msdb. Here, the only trace is the snapshot itself on the ZFS side.
• Point-in-time recovery is not covered. The snapshot alone brings the database back to the moment it was taken. For PITR, WAL archiving would still be required on top of it.

Conclusion

• The snapshot backup and restore model of the SQL Server series applies to PostgreSQL (no I/O freeze, no metadata backup).
• One important rule: data files and WAL must reside on the same zvol so the snapshot is atomic.
• A 146 GB database was restored in a few seconds and in less than 20 seconds under active load, WAL replay included.

Thank you. Amine Haloui

L’article PostgreSQL Snapshot Backup and Restore with Proxmox ZFS (4/4) est apparu en premier sur dbi Blog.

After creating the Ticket, I connect to the instance and see that the msdb system database is in a Suspect State:

After taking a deep breath because I don’t see that every day, I begin my investigation by asking the state with the system view sys.databases first:

msdb is really in a suspect mode…

Now, to begin the analysis, I go to read the error log.:

The Error Log give me the error that I comments in SSMS query to read it better:

Error: 3314, Severity: 21, State: 1.
During undoing of a logged operation in database ‘msdb’, an error occurred at log record ID (821931:128397:5).
Typically, the specific failure is logged previously as an error in the Windows Event Log service. Restore the database or file from a backup, or repair the database.

If you have this error message, the best is to restore the msdb;
As the information of the backups is in the msdb, I cannot do a right-click to restore the database…  

I looked for the latest backup in the error log and find it:

But I also sse that the checkdb before each backup give 12 errors…

My question is at this point: Wasn’t there an alert saying that the database was corrupted?

As the users are blocked, I need to react quickly.
This is why, I do my restore and a CHECKDB with  REPAIR_ALLOW_DATA_LOSS:

RESTORE DATABASE msdb FROM DISK = 'xxx\msdb\msdb_fullbackup_xxxx20260801xxxx.BAK' WITH REPLACE;
GO
 ALTER DATABASE msdb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
GO
DBCC CHECKDB ('msdb', REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS, ALL_ERRORMSGS;
GO
ALTER DATABASE msdb SET MULTI_USER;

The commands completed successfully! YES!

The database msdb is online and working fine. SUPER!

To be sure, I go in the errorlog to see the restore and the checkdb with REPAIR_ALLOW_DATA_LOSS:

The restore was successful and in the checkDB with can see 12 errors and repaired 12 errors.

If I will investigate deeper, I will go the dump file and analyse it.

I finally managed to sort out and resolve the critical alert on monday morning.
Users can connect and read their tables again…
The customer knows why he comes to us for an SLA…

A good victory for a monday morning in our support SLA dbi-services!
See you soon for sharing again customer cases!

L’article SQL Server:  msdb is in Suspect State est apparu en premier sur dbi Blog.

We turn up on a Monday morning and there are alerts from the weekend saying

[MSSQL] – Instance Dark_Vador-Data(F:): Disk space in GB on server Death_Star is critical

Info: NOT OK – Free space: 20 MB out of 200 GB

The storage is almost full and:
– no projects were deployed over the weekend
– no exceptional data loads were scheduled
– nobody works at the weekend

A strange phenomenon to come across on a Monday morning….

On closer inspection, the findings are surprising; we can see that a database within the instance has been undergoing regular autogrowth for weeks…

Every day, there are several regular autogrowth on the data file for this database until the saturation of the disk.

Who is the culprit?

Not the business data, but one or more tables named xxxxLog or Historyxxx, or the combination of both HistoryLog!

PS: It’s an example but in the reality before the weekend, we will have Warnings and in the majority of the cases we react before the critical alert of course!

Data created to diagnose a problem, track an activity or maintain a history, which is then never cleaned up or has a default retention period that is too long and unsuitable for the context, is not always easy to detect.

On the other hand, it’s always easier to expand disk capacity without too much effort, and everything runs smoothly in an ideal world…

We, however, are Green SQL Server DBA; we’ll look deeper.
Our curiosity will lead us to challenge the business on these points.

The AUDIT

First, you need to know the autogrowth settings for these databases and how the data file growth.
Here is a simple script that shows the settings:

SELECT
    DB_NAME(database_id) AS DatabaseName,
    name AS LogicalFileName,
    type_desc AS FileType,
    physical_name AS PhysicalFileName,
    size / 128.0 AS CurrentSizeMB,
    CASE
        WHEN is_percent_growth = 1
            THEN CAST(growth AS VARCHAR(10)) + ' %'
        ELSE CAST(growth / 128 AS VARCHAR(10)) + ' MB'
    END AS AutoGrowth,
    max_size,
    CASE
        WHEN max_size = -1 THEN 'Unlimited'
        ELSE CAST(max_size / 128 AS VARCHAR(20)) + ' MB'
    END AS MaxSize
FROM sys.master_files where type_desc='ROWS' AND database_id > 4
ORDER BY
    DatabaseName,
    FileType,
    LogicalFileName;

Like for my precedent blog “Green SQL Server DBA Tips  #1 – Are unnecessary indexes cluttering up your database?”, I use the Dynamics database as example:

In my script, I exclude all system databases and the Log File to be concentrated to the Data File.

Now, I need to have the data file growth. This is through the default trace file and the event 92:

DECLARE @TraceFile NVARCHAR(500);

SELECT @TraceFile = path
FROM sys.traces
WHERE is_default = 1;

IF @TraceFile IS NULL
BEGIN
    RAISERROR('Default Trace is disabled.',16,1);
    RETURN;
END

;WITH GrowthEvents AS
(
    SELECT
        DB_NAME(DatabaseID) AS DatabaseName, FileName, CAST(StartTime AS DATE) AS GrowthDate,
        DATEPART(YEAR, StartTime) AS YearNum, DATEPART(WEEK, StartTime) AS WeekNum,
        DATEPART(MONTH, StartTime) AS MonthNum, IntegerData * 8.0 / 1024 AS GrowthMB
    FROM fn_trace_gettable(@TraceFile, DEFAULT)
    WHERE EventClass IN (92) -- 92 is data growth
)

SELECT DatabaseName,FileName,GrowthDate,COUNT(*) AS GrowthEvents,SUM(GrowthMB) AS TotalGrowthMB
FROM GrowthEvents GROUP BY DatabaseName,FileName,GrowthDate ORDER BY DatabaseName, GrowthDate

As I don’t have any growth on the dynamicsBC database, I switch to the M-Files database for my example:

We have here some daily growth…

Go to see if we have historical or log tables in M-Files with this simple script who give us  the table and the row count and MB:

SELECT s.name AS Schema_Name,t.name AS Table_Name,
    SUM(p.rows) AS Row_Count, CAST(SUM(a.total_pages) * 8.0 / 1024 AS DECIMAL(18,2)) AS SizeMB
FROM sys.tables t INNER JOIN sys.schemas s     ON t.schema_id = s.schema_id
INNER JOIN sys.indexes i ON t.object_id = i.object_id
INNER JOIN sys.partitions p ON i.object_id = p.object_id  AND i.index_id = p.index_id
INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id
WHERE t.name LIKE 'Histor%' OR t.name LIKE '%Log'
GROUP BY s.name, t.name ORDER BY SizeMB DESC;

The ANALISYS

In my case, we can see that we have 383550 lignes (19.73Mb) on the table named OBJECTTYPECHANGELOG for a M-Files database.

It’s only ~20MB here but in few months perhaps 20 GB or more…

As a Green SQL Server DBA, I will contact the owner of M-Files project otherwise don’t forget that dbi services has also the expertise in ECM tools like M-Files.

The IMPACT

Like in my precedent post, we will have the same impacts

The first impact is on DML INSERT operations.

With every insertion, the log or historical information is written to the table and update perhaps also index and at the end is never used by a query.

The second impact is the storage and the backup

If the log table represents ~20 GB of the database.

It’s 20 GB backed up unnecessarily, 20 GB restored unnecessarily and, above all, 20 GB stored unnecessarily and just for one environment.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s more 100 GB!

The third impact is on the maintenance plan

The following operations must also handle the log/historical tables:
– Rebuild Index
– Reorganise Index
– CheckDB
– Backups
– Restores

Each growth of log/historical tables extends the maintenance windows…

The ADVISE

You really need to be careful before deleting anything in these table.

Don’t Truncate the table because you see it!

First see with the vendor if a purge or a retention parameter exists through the application. 

In my case with M-Files and the table OBJECTTYPECHANGELOG, in the administration interface, you can configure the retention. Go to challenge your M-Files team on it!

Finally, for every large log or historical table, you need to ask yourself the following questions:
– Who uses it?
– How often?
– Is there a legal obligation?
– What retention period is required?
– Can it be archived?
– Can it be purged?

The Green SQL Server DBA Score

Like for my precedent post, I setup a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

A traditional SQL Server DBA or system administrator will see a data disk full and says:

“How much GB need we to extend the disk?”

A Green SQL Server DBA sees a data disk full and says:

“Why is my disk full and what can I do?”

Just like adding a larger trunk doesn’t make a car more efficient, adding more GB doesn’t make a database healthier.

The greenest SQL Server DBA is not the one with the largest disks but it’s the one that keeps only data that continues to deliver value.

Think about it and begin now to see if you have cases!

See you soon for the next one! 

L’article Green SQL Server DBA Tips #2 – When logs and history records cause your database to growth and crash est apparu en premier sur dbi Blog.

When discussing SQL Server performance, we often hear our performance tool say, ‘An index is missing’, but never ‘There are too many indexes’.

However, in many databases, certain indexes are never used by business queries and they are maintained with every INSERT, UPDATE or DELETE operation.

The question is really simple:

• How much does an unused index actually cost?
• Why is an unused index a problem?

Every time there is a change to the data, SQL Server must also update all the relevant indexes, and that is not free.

As an example, I’m going to use a widely recognised database: the famous DynamicsBC.

The AUDIT

The first step in any study is always to carry out an inventory to gather all the data required for the analysis. I need the size of the database, size of each data file, size of the data and size of the indexes

I will use a query using the DMV: sys.dm_db_partition_stats

USE DynamicsBC;
GO

SELECT
DB_NAME() AS DatabaseName,
CAST(SUM(reserved_page_count) * 8.0 / 1024 AS DECIMAL(18,2)) AS TotalSizeMB,
CAST(SUM(used_page_count) * 8.0 / 1024 AS DECIMAL(18,2)) AS UsedSizeMB,
CAST((SUM(reserved_page_count) - SUM(used_page_count)) * 8.0 / 1024 AS DECIMAL(18,2)) AS FreeSizeMB,
CAST(SUM(
CASE
WHEN index_id < 2 THEN in_row_data_page_count
+ lob_used_page_count
+ row_overflow_used_page_count
ELSE 0
END
) * 8.0 / 1024 AS DECIMAL(18,2)) AS DataSizeMB,
CAST(SUM(
CASE
WHEN index_id >= 2 THEN used_page_count
ELSE 0
END
) * 8.0 / 1024 AS DECIMAL(18,2)) AS IndexSizeMB
FROM sys.dm_db_partition_stats;
GO

If we look at the ratio between the indexes and the data, we get:

64966.91/92898.87 × 100 = ~70 %

A ratio exceeding 50–60 % often requires an analysis of unused indexes, index overlap and compression.

YES, we are good with this database!

Now, let see if we have unused Indexes I use a script that I created years ago for DWH audit:

/******************************************************************************************/
--Summary
/******************************************************************************************/
USE [master];

DECLARE @sqlcmd NVARCHAR(max);
DECLARE @ExcludeFlag BIT = 0;
DECLARE @IncludeDB VARCHAR(1000);
DECLARE @ExcludeDB VARCHAR(1000);


SET  @IncludeDB = 'DynamicsBC'

SELECT @ExcludeFlag = 0;
--SELECT @ExcludeDB = 'distribution,master,model,msdb,tempdb';
 

DECLARE @InDBList TABLE (indb SYSNAME);
DECLARE @ExDBList TABLE (exdb SYSNAME);
DECLARE @dbList TABLE (dbID INT NOT NULL PRIMARY KEY, dbName SYSNAME NOT NULL);


IF (@ExcludeFlag = 0)
BEGIN
	INSERT INTO @InDBList SELECT * FROM string_split(@IncludeDB,',');
	INSERT INTO @dbList (dbID, dbName) 
		SELECT d.database_id, d.name 
		FROM sys.databases d 
		WHERE d.name IN (SELECT indb FROM @InDBList);
END
ELSE
BEGIN
--	INSERT INTO @ExDBList SELECT * FROM string_split(@ExcludeDB,',');
	INSERT INTO @dbList (dbID, dbName) 
		SELECT d.database_id, d.name 
		FROM sys.databases d 
--		WHERE d.name NOT IN (SELECT exdb FROM @ExDBList);
--		WHERE d.name NOT IN ('distribution','master','model','msdb','tempdb');

END

/*********************************************************************
Creation of the result table variable to store the information
--*********************************************************************/

IF (SELECT OBJECT_ID('tempdb.dbo.#dbUnUsedIndex')) IS NOT NULL  
	DROP TABLE dbo.#dbUnUsedIndex
CREATE TABLE #dbUnUsedIndex (
	dbID INT NOT NULL,
	dbName SYSNAME NOT NULL,
	objName SYSNAME NULL,
	idxName SYSNAME NULL,
	idxID INT NULL,
	UserSeek BIGINT NULL,
	UserScans BIGINT NULL,
	UserLookups BIGINT NULL,
	UserUpdates BIGINT NULL,
	nbRows BIGINT NULL,
	dropStatement NVARCHAR(1000) NULL)

DECLARE @dbID INT;
DECLARE @dbName SYSNAME;
DECLARE cur_dblst CURSOR 
  LOCAL STATIC READ_ONLY FORWARD_ONLY
FOR 
SELECT dbID, dbName
FROM @dbList;

OPEN cur_dblst
FETCH NEXT FROM cur_dblst INTO @dbID, @dbName
WHILE @@FETCH_STATUS = 0
BEGIN 
    --Do something with Id here

	SELECT @sqlcmd = N'
		USE [' + @dbName + ']'
		+ ' SELECT
			@dbID
			, @dbName			
			, o.name AS objName
			, i.name AS idxName
			, i.index_id AS idxID
			, dm_ius.user_seeks AS UserSeek
			, dm_ius.user_scans AS UserScans
			, dm_ius.user_lookups AS UserLookups
			, dm_ius.user_updates AS UserUpdates
			, p.TableRows AS nbRows
			, ''DROP INDEX '' + QUOTENAME(i.name)
			+ '' ON '' + QUOTENAME(s.name) + ''.''
			+ QUOTENAME(OBJECT_NAME(dm_ius.OBJECT_ID)) AS ''dropStatement''
		FROM sys.dm_db_index_usage_stats dm_ius
			INNER JOIN sys.indexes i ON i.index_id = dm_ius.index_id AND dm_ius.OBJECT_ID = i.OBJECT_ID
			INNER JOIN sys.objects o ON dm_ius.OBJECT_ID = o.OBJECT_ID
			INNER JOIN sys.schemas s ON o.schema_id = s.schema_id
			INNER JOIN (SELECT SUM(p.rows) TableRows, p.index_id, p.OBJECT_ID
			FROM sys.partitions p GROUP BY p.index_id, p.OBJECT_ID) p
			ON p.index_id = dm_ius.index_id AND dm_ius.OBJECT_ID = p.OBJECT_ID
		WHERE OBJECTPROPERTY(dm_ius.OBJECT_ID,''IsUserTable'') = 1
			AND dm_ius.database_id = DB_ID()
			AND i.type_desc = ''nonclustered''
			AND i.is_primary_key = 0
			AND i.is_unique_constraint = 0
		ORDER BY (dm_ius.user_seeks + dm_ius.user_scans + dm_ius.user_lookups) ASC; '

	INSERT INTO #dbUnUsedIndex (
					dbID,
					dbName,
					objName,
					idxName,
					idxID,
					UserSeek,
					UserScans,
					UserLookups,
					UserUpdates,
					nbRows,
					dropStatement)
	EXEC sp_executesql  @sqlCmd, N'@dbID INT, @dbName SYSNAME', @dbID, @dbName

    FETCH NEXT FROM cur_dblst INTO @dbID, @dbName
END
CLOSE cur_dblst
DEALLOCATE cur_dblst

SELECT Count(*) FROM #dbUnUsedIndex

SELECT * FROM #dbUnUsedIndex
ORDER BY dbName, objName


 IndexInfo;
GO

For the total number of Indexes, we use this simple query:

SELECT COUNT(*) AS TotalIndexes
FROM dynamicsBC.sys.indexes i
INNER JOIN sys.tables t
ON i.object_id = t.object_id
WHERE i.index_id > 0;

We have 998 unused Index for a total of 19494 Index. This is 5% of the Indexes.
In this case, we will not win a lot. 5% is good but we can have a look because we never known…

The IMPACT

The first impact is on DML INSERT and UPDATE operations.

INSERT

With every insertion, the data is written to the table and several index structures must also be updated, even when one of these indexes is never used by a query.

So we have the following sequence:

INSERT –> write to the table –> update index 1  –> update index 2  –> update index 3  –> update index 4 ….

This is even more costly for this command because, for each operation, the sequence is as follows:

UPDATE

UPDATE –> Delete the old index entry + Create the new entry for each index

The result of all of this is more I/O, more CPU usage and also more entries in the Transaction Log

On heavily used transactional tables, a few unused indexes can account for several per cent of additional load.

The second impact is the storage and the backup

If the unused index represents ~100B of the database.

If 100 GB of indexes are never used, that’s 100 GB backed up unnecessarily, 100 GB restored unnecessarily and, above all, 100 GB stored unnecessarily and just for one environment.

If you have Dev, Test, PreProd & Prod and all in HA… I let you do the calculation but it’s near 1 TB!

The third impact is on the maintenance plan

The following operations must also handle unused indexes:
– Rebuild Index
– Reorganise Index
– CheckDB
– Backups
– Restores

Each additional index extends the maintenance windows…

The ADVISE

You really need to be careful before deleting anything, as an unused index is not necessarily useless.

You should check several factors before you delete unused indexes:
– Full business cycle
– End/Begin of month
– Annual processing
– Reporting
– ETL
– SQL Agent jobs

I generally recommend observing the index between 1 and 3 months before deleting it and without restart. A restart will reset the DMV used for the stats… And always keep the script for recreating the index to hand!

The Green SQL Server DBA Score

Just for fun, we’re going to set up a ‘SQL Server DBA Score’ to use for my tips on the subject:

Conclusion

An unused index is a bit like car that never gets driven:
it takes up space;it costs money;it requires maintenance;but it produces no value.

The Green SQL Server DBA doesn’t just seek to add indexes.
They aim to retain only those that deliver measurable business value.

See you soon for the next one!

L’article Green SQL Server DBA Tips  #1 – Are unnecessary indexes cluttering up your database? est apparu en premier sur dbi Blog.

Your applications still need one stable address to connect to, and they need writes to reach the primary while reads spread across replicas. HAProxy handles that routing, with a floating IP from Keepalived in front of it.

How the tools work together

Three components stands between your application and the database.

Patroni manages replication and failover, and it runs an agent on every PostgreSQL node. Each agent exposes a small REST API (port 8008 by default) that reports that node’s role.

HAProxy accepts client connections and forwards them to the right node. It asks Patroni’s REST API which node is the primary and which are replicas, then sends each connection to a matching node.

Keepalived publishes a virtual IP that floats between your HAProxy hosts using VRRP. Your application connects to the VIP, so one HAProxy host going down doesn’t take the whole entry point with it.

Your application talks to the VIP. Keepalived points the VIP at a live HAProxy. HAProxy forwards the connection to whichever PostgreSQL node Patroni reports as healthy for that role.

The health-check method

HAProxy checks one port and routes to another.

Patroni’s REST API returns an HTTP status that depends on the node’s role:

• GET / returns 200 only on the leader (the primary). A non-leader node returns 503.
• GET /primary is the explicit name for the same leader check.
• GET /replica returns 200 only on a running replica.
• GET /read-only returns 200 on the primary or a replica, any node that can serve a read.

In our case, HAProxy runs its health check against the API port (8008) and reads that status code, then forwards the SQL connection to the database port (5432). A node receives traffic only when its API answers 200 for the role that listener cares about. Point a listener’s check at / and it follows the primary. Point it at /replica and it follows the replicas. Patroni promotes a new leader, the status codes change, and HAProxy moves traffic to match within a couple of health-check cycles.

A first and simple working configuration

A two-node setup with 10.5.5.147 and 10.5.5.148 looks like this. One listener handles writes, the other handles reads.

listen PG1
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

listen PG1_ro
    bind *:5001
    option httpchk GET /replica
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

This example runs PostgreSQL on port 5432 and the Patroni API on 8008, so swap in whatever ports your deployment uses (the defaults are 5432 and 8008).

Line by line:

• bind *:5000 and bind *:5001 are the two addresses your applications connect to. Send writes to 5000 and reads to 5001.
• option httpchk (with no path) on the first listener checks Patroni’s root endpoint. Only the leader answers 200, so HAProxy sends port 5000 traffic to the current primary.
• option httpchk GET /replica on the second listener checks the replica endpoint, so HAProxy sends port 5001 traffic to a replica.
• http-check expect status 200 tells HAProxy that 200 means healthy and anything else means down.
• inter 3s fall 3 rise 2 checks every 3 seconds, marks a server down after 3 failures, and brings it back after 2 successes.
• on-marked-down shutdown-sessions kills existing connections to a server the instant HAProxy marks it down, so clients reconnect and get rerouted instead of hanging on a dead node.
• check port 8008 is the trick in action: health checks hit the Patroni API on 8008 while HAProxy forwards traffic to PostgreSQL on 5432.
• maxconn 100 limit connections per server so you don’t exhaust PostgreSQL’s connection slots.

For a primary plus one or more replicas, this routes writes and reads to the right node and survives a failover.

The failure mode hiding in the read path

Imagine a two-node cluster: one primary, one replica. The replica goes down. Maybe it crashed, maybe Patroni is mid-switchover and no standby exists for a few seconds.

Your read traffic hits port 5001. That listener marks a server up only when GET /replica returns 200, and right now no node is a replica. HAProxy has zero usable servers in the pool, so it refuses the connection. Read queries start failing.

The primary is up the entire time, and it can serve those reads. Your config won’t send them there, because you told the read listener to look for replicas and nothing else. You’ve turned a degraded cluster that could still serve reads into a read outage. You feel this most on small clusters, and each failover passes through a window where the old primary becomes a replica and no standby is available yet. In the worst case, your replica is down, and one of your application is connecting to port 5001, resulting in errors.

The fix: fall back to the primary

Send reads to the primary when the read listener runs out of replicas, instead of dropping them.

listen PG1
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

listen PG1_ro
    bind *:5001
    option httpchk GET /replica
    http-check expect status 200
    use_backend PG1_ro_leader if { nbsrv(PG1_ro) eq 0 }
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

backend PG1_ro_leader
    option httpchk GET /primary
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_10.5.5.147_5432 10.5.5.147:5432 maxconn 100 check port 8008
    server postgresql_10.5.5.148_5432 10.5.5.148:5432 maxconn 100 check port 8008

This new line carries the whole fix:

use_backend PG1_ro_leader if { nbsrv(PG1_ro) eq 0 }

nbsrv(PG1_ro) counts the usable servers in the PG1_ro pool, which here means the number of available replicas, since those servers pass the check only when GET /replica returns 200. While at least one replica is up, the count stays above zero, the condition is false, and reads stay on the replicas. The moment the last replica drops, nbsrv(PG1_ro) hits zero, the condition fires, and HAProxy diverts reads to the PG1_ro_leader backend.

That backend health-checks with GET /primary, so the only server it counts as up is the current primary. HAProxy sends reads to the primary until a replica returns, then shifts them back to the replica pool once a replica passes its check again.

Three names have to agree for this to work. The backend you define (backend PG1_ro_leader), the backend you route to (use_backend PG1_ro_leader), and the pool you count (nbsrv(PG1_ro)) all reference the real section names. Drop in a stale name from an earlier version and HAProxy either refuses to start or counts the wrong pool.

The /read-only shortcut and what it costs

Patroni offers GET /read-only, which returns 200 on the primary and the replicas alike. Point the read listener there and both the primary and the replicas serve reads, no fallback backend needed.

The cost is read load on the primary even when your replicas are healthy and idle. The fallback approach keeps reads off the primary until the replicas are gone, then leans on it as a safety net. You protect the primary’s write capacity during normal operation and still keep reads alive during a replica outage.

To keep lagging replicas out of the read pool, Patroni accepts a threshold on the replica check, for example GET /replica?lag=10MB, which fails any replica more than 10 MB behind. Pair that with the fallback and HAProxy drops the lagging replicas from rotation while reads still have somewhere to go.

Keepalived: removing HAProxy as a single point of failure

One HAProxy host fronting the cluster moves the single point of failure up a layer. Run HAProxy on two hosts and let Keepalived float a virtual IP between them with VRRP. Your application connects to the VIP, and whichever HAProxy holds it answers.

A minimal keepalived.conf on the primary HAProxy host:

vrrp_script chk_haproxy {
    script "killall -0 haproxy"   # succeeds while the haproxy process is alive
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass changeme
    }
    virtual_ipaddress {
        10.0.0.1
    }
    track_script {
        chk_haproxy
    }
}

The second HAProxy host runs the same file with state BACKUP and a lower priority (100). Both advertise over VRRP, and the higher priority holds the VIP. chk_haproxy runs every two seconds. If HAProxy dies on the active host, its priority drops and the backup takes the VIP, so an HAProxy crash on one host no longer takes the entry point down with it.

Point your applications at 10.0.0.1:5000 for writes and 10.5.5.100:5001 for reads. Your applications never see which physical HAProxy does the work.

Summary

Patroni keeps the cluster healthy and picks the leader. HAProxy turns Patroni’s REST API into routing, sending writes to the primary and reads to the replicas by health-checking the API port while forwarding to the database port. The naive read-only listener drops reads when the last replica goes down, even though the primary could serve them. Adding use_backend ... if { nbsrv(...) eq 0 } with a primary-checking backend closes that gap, and a lag threshold on the replica check keeps stale standbys out of rotation. Keepalived puts a floating VIP in front of two HAProxy instances so the proxy layer survives a host failure too.

Writes reach the primary and reads spread across the replicas. Reads stay up as long as one node in the cluster is alive.

Let me know if you find any improvements to this configuration

L’article Highly Available, Load-Balanced PostgreSQL with Patroni, HAProxy, and Keepalived est apparu en premier sur dbi Blog.

The interesting and challenging aspect of this setup is ensuring that the encryption and decryption remains intact after a failover.

This challenge originates from SQL Server’s encryption hierarchy. In a typical setup, the Service Master Key (SMK) on server level protects the Database Master Key (DMK) on database level. The Database Master Key, in turn, protects certificates and asymmetric keys, while certificates are commonly used to protect symmetric keys. These symmetric keys may then be used by the application to encrypt and decrypt sensitive data.

When a failover to a secondary replica occurs, the Service Master Key (SMK) is no longer the same as on the previous primary replica. Since the Database Master Key (DMK) is typically encrypted by the local Service Master Key, SQL Server may no longer be able to open the DMK transparently after the failover. As a result, any encryption objects that depend on the DMK, such as certificates and symmetric keys, may become inaccessible, potentially causing application failures when encrypted data needs to be read or written.

Let’s now explore the built-in approaches in SQL Server to ensure that the encryption hierarchy remains functional across replicas and that applications continue to operate seamlessly after a failover.

First, let’s create a test database and a Database Master Key:

-- Create Demo DB
CREATE DATABASE TestDMK;
GO
--Backup DB
BACKUP DATABASE TestDMK
TO DISK = 'C:\SQLServer_mnt\BACKUP\TestDMK.bak';
go
-- Create Database Master Key
CREATE MASTER KEY
ENCRYPTION BY PASSWORD = 'MyVeryStrongPassword_123!';
GO
-- Check Master Key
SELECT DB_NAME() as DB, *
FROM sys.symmetric_keys
WHERE name = '##MS_DatabaseMasterKey##';
GO

It is worth noting that you can verify whether a Database Master Key is also encrypted by the Service Master Key by querying the sys.databases catalog view. When a Database Master Key is created, SQL Server typically adds encryption by the Service Master Key automatically. This behaviour can be changed by explicitly removing the Service Master Key encryption.

select is_master_key_encrypted_by_server, * from sys.databases where name = 'TestDMK'

With the Database Master Key in place, let’s create a certificate and a symmetric key that will be used throughout the remainder of this demonstration.

CREATE CERTIFICATE [mainDBCert] --> This cert is going to be encrypted by the dmk
WITH SUBJECT = 'test Cert';
go
CREATE SYMMETRIC KEY mainKey
WITH ALGORITHM = AES_256
ENCRYPTION BY CERTIFICATE mainDBCert;
go
SELECT * FROM sys.certificates;
SELECT * FROM sys.symmetric_keys WHERE name <> '##MS_DatabaseMasterKey##';
SELECT * FROM sys.symmetric_keys;
go

To demonstrate the encryption and decryption process, let’s create a simple table that will contain sensitive information and insert some data.

--create table with sensitive data
CREATE TABLE CreditCards
(
    Id int,
    CreditCardNr varbinary(max),
    CVC varbinary(max),
    ExpirationDate varbinary(max)
);
go
--open symmetric key
OPEN SYMMETRIC KEY mainKey
DECRYPTION BY CERTIFICATE mainDBCert;
--insert sensitive data
INSERT INTO CredîtCards
VALUES
(
    1,
    EncryptByKey(Key_GUID('mainKey'), '1234-5678-9012-3456'),
    EncryptByKey(Key_GUID('mainKey'), '123'),
    EncryptByKey(Key_GUID('mainKey'), '1233')
);
CLOSE SYMMETRIC KEY mainKey;
GO
Select * from CreditCards

If we query the table without opening the symmetric key and decrypting the values, we can see that we won’t see the data.

--select data without decryption
SELECT
    Id,
    CONVERT(varchar(50), CreditCardNr) AS CreditCardNr,
    CONVERT(varchar(10), CVC) AS CVC,
    CONVERT(varchar(10), ExpirationDate) AS ExpirationDate
FROM CreditCards;

However, if we open the symmetric key first and use it to decrypt the data, we are able to retrieve the original sensitive information:

OPEN SYMMETRIC KEY mainKey
DECRYPTION BY CERTIFICATE mainDBCert;
GO
SELECT
    Id,
    CONVERT(varchar(50), DecryptByKey(CreditCardNr)) AS CreditCardNr,
    CONVERT(varchar(10), DecryptByKey(CVC)) AS CVC,
    CONVERT(varchar(10), DecryptByKey(ExpirationDate)) AS ExpirationDate
FROM CredîtCards;
GO
CLOSE SYMMETRIC KEY mainKey;
GO

Now let’s add our database to an Always On Availability Group to provide high availability and disaster recovery. If you follow the Availability Group wizard, you may notice that the wizard asks you to enter a password. What the wizards is asking for here is the password of the Database Master Key (DMK). The DMK protects the certificate in our encryption hierarchy, and the certificate, in turn, protects the symmetric key used to encrypt and decrypt the sensitive data.

On the right-hand side of the wizard, there is a small and easily overlooked text box where you can enter the Database Master Key password.

And now comes the tricky – and at the same time somewhat surprising – part. If you use automatic seeding, which is my preferred option whenever the database size allows it, the process will skip the part where the Master Key password is verified and applied on the secondary replicas.

As a result, when a failover occurs, SQL Server would no longer be able to open the Database Master Key (DMK) transparently on the new primary replica. Consequently, the application would no longer be able to encrypt or decrypt data unless the Database Master Key is explicitly opened first with the password.

In such a scenario, the issue can be resolved by storing the Database Master Key password as a credential on each replica using the sp_control_dbmasterkey_password stored procedure. SQL Server can then use this credential to automatically open the Database Master Key after a failover, allowing the encryption hierarchy to remain intact and ensuring that certificates, symmetric keys, and encrypted data remain accessible transparently.

!!Note that you must do that on each replica!!

sp_control_dbmasterkey_password @db_name = N'TestDMK'
    , @password = N'MyVeryStrongPassword_123!'
    , @action = N'add';
select * from sys.master_key_passwords;
select * from sys.credentials;

Once the credential has been created, SQL Server is again able to open the Database Master Key transparently. As a result, the application can continue to encrypt and decrypt data without having to explicitly open the Master Key with the password.

There is also an option by which SQL Server automatically creates the Database Master Key credential across all replicas when a database is joined to an Availability Group. This happens when “Full Database Backup and Log Backup” is selected as the initial data synchronization option in the Availability Group wizard.

To demonstrate this, let’s remove the previously created credential using the same stored procedure.

sp_control_dbmasterkey_password @db_name = N'TestDMK'
    , @password = N'MyVeryStrongPassword_123!'
    , @action = N'drop';
select * from sys.master_key_passwords;
select * from sys.credentials;

Now, let’s select “Full Database and Log Backup” as the initial data synchronization option in the Availability Group wizard.

As you can see, the process is no longer skipping the Database Master Key password validation step.

In the summary, you can see that the process automatically created the Database Master Key password credential on each replica.

If you check the credentials on each replica afterwards, you can verify that they were created automatically.

As the Database Master Key password credential has been added to all replicas, SQL Server can transparently open the DMK after a failover. As a result, the application can continue to encrypt and decrypt data without having to explicitly open the Database Master Key.

But one question still puzzles me: why the same operation isn’t performed automatically when automatic seeding is used?

Honestly, I have no clue yet. If you know more let me know it in the comments section. I’d love to learn more about it.

Thanks for reading – Hocine 😉

L’article SQL Server Always On Availability Groups and Database Master Keys: A Hidden Failover Pitfall est apparu en premier sur dbi Blog.

For the first time, I joined some days ago the Data Point Prague . The biggest International Data Conference in Prague. A two-day event dedicated to advancing knowledge in data technologies.
Thanks to dbi-services to let me the possibility to attend this event.

Thursday, May 28, 2026

The first day was dedicated to pre-conference Workshops. I choose the workshop “Performance optimization by identifying and correcting bad SQL code” with Uwe Ricken as a speaker.
During the full day, Uwe shown us through different realistic scenarios how to identify, analyze and optimize inefficient SQL Codes with usage of tools like Query Store, Windows Admin Center or even Perfmon.

Friday, May 29, 2026

The second day started with the Keynote session by Jorge Docampo Carro, Senior Program Manager at Microsoft.
He shown us that AI is no longer just used to create peace of codes, it is becoming an active collaborator in how Spark jobs, lakehouses, and data pipelines are designed, implemented, and operated.
Moreover Visual Studio Code is appearing as a preferred workspace for Fabric data engineering — bringing notebooks, Spark job definitions, environments, and lakehouse artifacts directly into the developer experience.

I then followed a very interesting but quite intensive session of Torsten Strauss named “A Deep Dive into Optimized Locking in SQL Server 2025”.
During this session Torsten talked about the introduction with SQL Server 2025 of Optimized Locking and Lock After Qualification (LAQ) to reduce contention and improve concurrency.
He explained that LAQ defers lock acquisition until rows are fully evaluated, meaning only qualifying rows are locked instead of locking during the scan phase. This significantly lowers the number and duration of locks, reducing blocking, deadlocks, and lock escalation in high-concurrency workloads.
He shows us this new Locking mechanism through different concrete examples. The improvements are especially impactful for large updates, selective queries, and OLTP systems with heavy parallel activity. Combined with proper indexing, transaction design, and features like RCSI, these enhancements enable higher throughput and more stable performance.

In the Afternoon I saw an interesting session from Uwe again about the Security Techniques for cross database access. It was a good refresh on how to grant access but keep the security at a high level to protect data. Discussions and examples turned around Synonyms, signed Stored Procedures and Trustworthy database options.

I joined also some sessions more related to PowerBi, Fabric or AI which were also really interesting even if I’m not an expert in those domains.

It was my first time at Data Point Prague #DataPointPrague but certainly not the last one as this conference is really well organized. The first full day workshop was really great and the second day with more than 30 sessions let you choose the ones you are interesting on.
Thank you very much for this event and see you next time

L’article Data Point Prague 2026 est apparu en premier sur dbi Blog.

The role of the API is deliberately limited: it receives a snapshot request, checks that the requested zvol is authorized, and then runs the zfs snapshot command on the Proxmox side. An allowlist is used to restrict the ZFS volumes that can be accessed. This prevents a REST call from being able to manipulate any dataset on the server.

With this approach, we can reproduce a behavior close to what an enterprise storage array provides, but using Proxmox and ZFS. It is important to note that Proxmox does not natively provide the same level of integration as Pure Storage for SQL Server snapshots. Pure Storage provides dedicated mechanisms and integrations. In our case, we need to build a specific orchestration layer. The REST API therefore acts as an adapter between SQL Server, which drives the snapshot backup workflow, and ZFS, which actually performs the storage-level snapshot.

Architecture

Here is a global overview of the architecture:

• SQL Server freezes the database I/Os
• SQL Server 2025 calls the internal REST API
• The REST API validates the request and checks the zvol allowlist
• The API triggers the ZFS snapshot on Proxmox
• The API returns the snapshot information to SQL Server
• SQL Server creates the metadata-only backup
• The database I/Os are released

REST API implementation

Under Proxmox, we install the required packages:

apt update
apt install -y python3-venv sudo openssl

We create a dedicated user:

useradd --system \
  --home /opt/sql-zfs-api \
  --shell /usr/sbin/nologin \
  sqlsnap

We create the following folders:

mkdir -p /opt/sql-zfs-api
mkdir -p /etc/sql-zfs-api

We declare the authorized zvol :

cat >/etc/sql-zfs-api/allowed-zvols <<'EOF'
sqlpool/pve/vm-302-disk-0
EOF

We create a root-only allowlist:

chown root:root /etc/sql-zfs-api/allowed-zvols
chmod 600 /etc/sql-zfs-api/allowed-zvols

Then we create the secured ZFS helper. This script is executed as root through sudo, but it rejects any dataset that is not defined in the allowlist.

cat >/usr/local/sbin/sql-zfs-helper <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

ALLOW_FILE="/etc/sql-zfs-api/allowed-zvols"
LOCK_FILE="/run/sql-zfs-helper.lock"

die() {
  echo "$*" >&2
  exit 1
}

exec 9>"$LOCK_FILE"
flock -n 9 || die "another snapshot operation is already running"

[[ -r "$ALLOW_FILE" ]] || die "allowlist not readable: $ALLOW_FILE"

mapfile -t ALLOWED_DATASETS < <(grep -Ev '^\s*(#|$)' "$ALLOW_FILE")

is_allowed() {
  local ds="$1"
  local allowed
  for allowed in "${ALLOWED_DATASETS[@]}"; do
    [[ "$ds" == "$allowed" ]] && return 0
  done
  return 1
}

valid_snapname() {
  [[ "$1" =~ ^[A-Za-z0-9_.:-]{1,120}$ ]]
}

ACTION="${1:-}"
shift || true

case "$ACTION" in
  snapshot)
    SNAPNAME="${1:-}"
    shift || true

    valid_snapname "$SNAPNAME" || die "invalid snapshot name: $SNAPNAME"
    [[ "$#" -ge 1 ]] || die "no zvol specified"
    [[ "$#" -le 8 ]] || die "too many zvols"

    SNAPSHOTS=()

    for DS in "$@"; do
      is_allowed "$DS" || die "dataset not allowed: $DS"
      /sbin/zfs list -H -t volume -o name "$DS" >/dev/null 2>&1 || die "zvol not found: $DS"

      FULLSNAP="${DS}@${SNAPNAME}"

      if /sbin/zfs list -H -t snapshot -o name "$FULLSNAP" >/dev/null 2>&1; then
        die "snapshot already exists: $FULLSNAP"
      fi

      SNAPSHOTS+=("$FULLSNAP")
    done

    /sbin/zfs snapshot "${SNAPSHOTS[@]}"
    /sbin/zfs hold sqlsnap "${SNAPSHOTS[@]}"

    printf '{"status":"ok","snapshots":['
    SEP=""
    for S in "${SNAPSHOTS[@]}"; do
      printf '%s"%s"' "$SEP" "$S"
      SEP=","
    done
    printf ']}\n'
    ;;

  list)
    /sbin/zfs list -H -t snapshot -o name -r sqlpool | grep '@sql_' || true
    ;;

  *)
    die "usage: sql-zfs-helper snapshot SNAPNAME ZVOL [ZVOL...]"
    ;;
esac
EOF

chown root:root /usr/local/sbin/sql-zfs-helper
chmod 750 /usr/local/sbin/sql-zfs-helper

We only allow the helper through sudo:

cat >/etc/sudoers.d/sql-zfs-helper <<'EOF'
sqlsnap ALL=(root) NOPASSWD: /usr/local/sbin/sql-zfs-helper *
EOF

chmod 440 /etc/sudoers.d/sql-zfs-helper
visudo -cf /etc/sudoers.d/sql-zfs-helper

We install the FastAPI API:

python3 -m venv /opt/sql-zfs-api/venv
/opt/sql-zfs-api/venv/bin/pip install fastapi "uvicorn[standard]"

We create the application file:

cat >/opt/sql-zfs-api/app.py <<'EOF'
import os
import re
import json
import socket
import secrets
import subprocess
from datetime import datetime, timezone
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field

API_KEY = os.environ.get("SQL_ZFS_API_KEY", "")
ALLOW_FILE = "/etc/sql-zfs-api/allowed-zvols"
SNAP_RE = re.compile(r"^[A-Za-z0-9_.:-]{1,120}$")

app = FastAPI(title="SQL ZFS Snapshot API", version="1.0.0")


class SnapshotRequest(BaseModel):
    database: str = Field(..., min_length=1, max_length=128)
    vmid: int = 302
    snapname: str = Field(..., min_length=1, max_length=120)
    zvols: list[str] = Field(..., min_length=1, max_length=8)


def load_allowed_zvols() -> set[str]:
    with open(ALLOW_FILE, "r", encoding="utf-8") as f:
        return {
            line.strip()
            for line in f
            if line.strip() and not line.strip().startswith("#")
        }


def check_api_key(x_sqlsnap_key: str | None) -> None:
    if not API_KEY:
        raise HTTPException(status_code=500, detail="API key not configured")

    if not x_sqlsnap_key:
        raise HTTPException(status_code=401, detail="missing API key")

    if not secrets.compare_digest(x_sqlsnap_key, API_KEY):
        raise HTTPException(status_code=403, detail="invalid API key")


@app.get("/health")
def health():
    return {
        "status": "ok",
        "host": socket.gethostname(),
        "utc": datetime.now(timezone.utc).isoformat(),
    }


@app.post("/v1/sql-zfs/snapshot")
def create_snapshot(
    req: SnapshotRequest,
    x_sqlsnap_key: str | None = Header(default=None, alias="x-sqlsnap-key"),
):
    check_api_key(x_sqlsnap_key)

    if not SNAP_RE.fullmatch(req.snapname):
        raise HTTPException(status_code=400, detail="invalid snapname")

    allowed = load_allowed_zvols()

    for zvol in req.zvols:
        if zvol not in allowed:
            raise HTTPException(status_code=403, detail=f"zvol not allowed: {zvol}")

    cmd = [
        "sudo",
        "/usr/local/sbin/sql-zfs-helper",
        "snapshot",
        req.snapname,
        *req.zvols,
    ]

    try:
        completed = subprocess.run(
            cmd,
            text=True,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            timeout=30,
            check=False,
        )
    except subprocess.TimeoutExpired:
        raise HTTPException(status_code=504, detail="zfs snapshot timeout")

    if completed.returncode != 0:
        raise HTTPException(
            status_code=500,
            detail={
                "error": completed.stderr.strip(),
                "stdout": completed.stdout.strip(),
            },
        )

    snapshots = [f"{zvol}@{req.snapname}" for zvol in req.zvols]

    return {
        "status": "ok",
        "database": req.database,
        "vmid": req.vmid,
        "snapname": req.snapname,
        "snapshots": snapshots,
        "media_description": "zfs|" + socket.gethostname() + "|" + ";".join(snapshots),
    }
EOF

chown -R root:root /opt/sql-zfs-api
chmod 755 /opt/sql-zfs-api
chmod 644 /opt/sql-zfs-api/app.py

We configure and generate the key:

APIKEY="$(openssl rand -hex 32)"
echo "$APIKEY"

We create the environment file:

cat >/etc/sql-zfs-api/sql-zfs-api.env <<EOF
SQL_ZFS_API_KEY=$APIKEY
EOF

chown root:root /etc/sql-zfs-api/sql-zfs-api.env
chmod 600 /etc/sql-zfs-api/sql-zfs-api.env

We need to save the generated key.

Next, we enable HTTPS. SQL Server sp_invoke_external_rest_endpoint calls HTTPS endpoints, and the documentation specifies that only HTTPS endpoints with TLS are supported.

openssl req -x509 -newkey rsa:4096 -sha256 -days 360 -nodes \
  -keyout /etc/sql-zfs-api/tls.key \
  -out /etc/sql-zfs-api/tls.crt \
  -subj "/CN=promox1" \
  -addext "subjectAltName=DNS:promox1,IP:192.168.1.110"

chown root:sqlsnap /etc/sql-zfs-api/tls.key /etc/sql-zfs-api/tls.crt
chmod 640 /etc/sql-zfs-api/tls.key
chmod 644 /etc/sql-zfs-api/tls.crt

The /etc/sql-zfs-api/tls.crt certificate must be imported into the Windows trusted root certification authorities on the SQL Server side. Otherwise, the HTTPS call may fail.

We create the systemd service:

cat >/etc/systemd/system/sql-zfs-api.service <<'EOF'
[Unit]
Description=SQL Server to ZFS Snapshot API
After=network-online.target
Wants=network-online.target

[Service]
User=sqlsnap
Group=sqlsnap
WorkingDirectory=/opt/sql-zfs-api
EnvironmentFile=/etc/sql-zfs-api/sql-zfs-api.env
ExecStart=/opt/sql-zfs-api/venv/bin/uvicorn app:app --host 0.0.0.0 --port 8443 --ssl-keyfile /etc/sql-zfs-api/tls.key --ssl-certfile /etc/sql-zfs-api/tls.crt
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now sql-zfs-api
systemctl status sql-zfs-api

We check the status of our API:

It is possible to call the API in PowerShell using Invoke-RestMethod with PowerShell 7:

$headers = @{
"Content-Type"  = "application/json"
"x-sqlsnap-key" = "MyKey"
}

$body = @{
database = "StackOverflow"
vmid     = 302
snapname = "StackOverflow_test010"
zvols    = @("sqlpool/pve/vm-302-disk-0")
} | ConvertTo-Json -Depth 5

Invoke-RestMethod `
-Uri "https://192.168.1.110:8443/v1/sql-zfs/snapshot" `
-Method Post `
-Headers $headers `
-Body $body `
-ContentType "application/json" `
-SkipCertificateCheck

This gives:

Test from SQL Server

A certificate was generated on Proxmox and it needs to be imported on the SQL Server host. In my case, it was located here:

I then imported it on Windows Server:

For testing purposes, I created something simple. On the SQL Server side, we can create a database that will be used to store our future stored procedure. This procedure will allow us to interact with the API. In my case, I created a database called dbi_tools:

This database will contain a credential. In our case, the DATABASE SCOPED CREDENTIAL is used to securely store the authentication information required to call the REST API from SQL Server. This allows us, for example, to protect the API key:

USE [dbi_tools]
GO

IF NOT EXISTS (
    SELECT 1
    FROM sys.symmetric_keys
    WHERE name = '##MS_DatabaseMasterKey##'
)
BEGIN
    CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'MyStrongPassword_%99';
END
GO

CREATE DATABASE SCOPED CREDENTIAL [https://192.168.1.110:8443/v1/sql-zfs/snapshot]
WITH
    IDENTITY = 'HTTPEndpointHeaders',
    SECRET = '{"x-sqlsnap-key":"MyAPIKey"}';
GO

We then create a stored procedure to encapsulate the code used to call the API:

USE dbi_tools;
GO

CREATE OR ALTER PROCEDURE dbo.usp_BackupDatabase_WithZfsSnapshot
    @DatabaseName sysname,
    @BackupDirectory nvarchar(4000) = N'D:\Backups\'
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @Url nvarchar(4000) =
        N'https://192.168.1.110:8443/v1/sql-zfs/snapshot';

    DECLARE @Vmid int = 302;

    DECLARE @ZvolsJson nvarchar(max) =
        N'["sqlpool/pve/vm-302-disk-0"]';

    DECLARE @Stamp varchar(20) =
        REPLACE(REPLACE(CONVERT(varchar(19), SYSUTCDATETIME(), 126), '-', ''), ':', '') + 'Z';

    DECLARE @SafeDbName nvarchar(128) =
        REPLACE(REPLACE(REPLACE(@DatabaseName, N' ', N'_'), N'[', N''), N']', N'');

    DECLARE @SnapName nvarchar(128) =
        CONCAT(N'sql_', @SafeDbName, N'_', @Stamp);

    DECLARE @BackupFile nvarchar(4000) =
        CONCAT(@BackupDirectory, N'\', @SafeDbName, N'_', @Stamp, N'.bkm');

    DECLARE @Payload nvarchar(max) =
    (
        SELECT
            @DatabaseName AS [database],
            @Vmid AS [vmid],
            @SnapName AS [snapname],
            JSON_QUERY(@ZvolsJson) AS [zvols]
        FOR JSON PATH, WITHOUT_ARRAY_WRAPPER
    );

    DECLARE @ReturnCode int;
    DECLARE @Response nvarchar(max);
    DECLARE @SnapshotList nvarchar(max);

    SELECT @SnapshotList =
        STRING_AGG(CONCAT([value], N'@', @SnapName), N';')
    FROM OPENJSON(@ZvolsJson);

    DECLARE @MediaDescription nvarchar(max) =
        CONCAT(N'zfs|promox1|', @SnapshotList);

    DECLARE @Sql nvarchar(max);

    BEGIN TRY
        SET @Sql =
            N'ALTER DATABASE ' + QUOTENAME(@DatabaseName) +
            N' SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON;';

        EXEC sys.sp_executesql @Sql;

        EXEC @ReturnCode = sys.sp_invoke_external_rest_endpoint
            @url = @Url,
            @method = N'POST',
            @headers = N'{"Content-Type":"application/json","Accept":"application/json"}',
            @payload = @Payload,
            @credential = [https://192.168.1.110:8443/v1/sql-zfs/snapshot],
            @timeout = 30,
            @response = @Response OUTPUT;

        IF @ReturnCode <> 0
        BEGIN
            DECLARE @Err nvarchar(max) =
                CONCAT(N'ZFS snapshot API failed. ReturnCode=', @ReturnCode, N' Response=', @Response);
            THROW 51001, @Err, 1;
        END;

        SET @Sql =
            N'BACKUP DATABASE ' + QUOTENAME(@DatabaseName) + N'
              TO DISK = @BackupFile
              WITH METADATA_ONLY,
                   FORMAT,
                   MEDIANAME = @MediaName,
                   MEDIADESCRIPTION = @MediaDescription,
                   NAME = @BackupName;';

        EXEC sys.sp_executesql
            @Sql,
            N'@BackupFile nvarchar(4000),
              @MediaName nvarchar(128),
              @MediaDescription nvarchar(max),
              @BackupName nvarchar(128)',
            @BackupFile = @BackupFile,
            @MediaName = @SnapName,
            @MediaDescription = @MediaDescription,
            @BackupName = @SnapName;

        SELECT
            @DatabaseName AS database_name,
            @SnapName AS zfs_snapshot_name,
            @SnapshotList AS zfs_snapshots,
            @BackupFile AS metadata_backup_file,
            @MediaDescription AS media_description,
            @Response AS api_response;
    END TRY
    BEGIN CATCH
        IF DATABASEPROPERTYEX(@DatabaseName, 'IsDatabaseSuspendedForSnapshotBackup') = 1
        BEGIN
            SET @Sql =
                N'ALTER DATABASE ' + QUOTENAME(@DatabaseName) +
                N' SET SUSPEND_FOR_SNAPSHOT_BACKUP = OFF;';

            EXEC sys.sp_executesql @Sql;
        END;

        THROW;
    END CATCH
END;
GO

We then call the stored procedure:

EXEC dbi_tools.dbo.usp_BackupDatabase_WithZfsSnapshot
    @DatabaseName = N'StackOverflow',
    @BackupDirectory = N'D:\Backups\';

The backup was generated :

References

sp_invoke_external_rest_endpoint

Thank you. Amine Haloui

L’article SQL Server Snapshot Backup and Restore with Proxmox ZFS – REST API with SQL Server 2025 (3/4) est apparu en premier sur dbi Blog.

To address this issue, it is possible to automate the solution with PowerShell. The idea is to automate the different operations involved in the snapshot backup and restore process.

We will use two scripts:

• One script to perform the backups and create the snapshots.
• One script to perform the restores.

Backup process

Here is how the backup process works:

• We connect to the corresponding SQL Server instance.
• We change the state of the database using ALTER DATABASE … SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON. At this point, the I/Os are frozen.
• We connect to the hypervisor through SSH.
• We create the snapshot.
• We back up the database using BACKUP DATABASE … WITH METADATA_ONLY.
• We change the state of the database using ALTER DATABASE … SET SUSPEND_FOR_SNAPSHOT_BACKUP = OFF. At this point, the I/Os are unfrozen.

Powershell implementation (backup)

Here is the code used to perform the backup:

param(
    [string]$SqlInstance = "VM-WS25-SQL2",
    [string]$Database    = "StackOverflow",
    [string]$BackupDir   = "D:\Backups",
    [string]$PveHost     = "192.168.1.110",
    [string]$PveUser     = "MyUser",
    [string[]]$Zvols     = @("sqlpool/pve/vm-302-disk-0")
)

$Timestamp = Get-Date -Format "yyyyMMddTHHmmss"
$SnapName  = "sql_${Database}_${Timestamp}"

$DbSafe = $Database.Replace("]", "]]")
$BackupFile = Join-Path $BackupDir "${Database}_${Timestamp}.bkm"

$ZfsSnapshots = $Zvols | ForEach-Object { "$_@$SnapName" }
$ZfsSnapshotArgs = $ZfsSnapshots -join " "

$MediaDescription = "zfs|$PveHost|$ZfsSnapshotArgs"

$BackupFileSql = $BackupFile.Replace("'", "''")
$MediaSql = $MediaDescription.Replace("'", "''")

$connString = "Server=$SqlInstance;Database=master;Integrated Security=True;TrustServerCertificate=True;Application Name=ZFS-TSQL-Snapshot;"
$conn = New-Object System.Data.SqlClient.SqlConnection $connString

function Invoke-SqlNonQuery {
    param([string]$Sql)

    $cmd = $conn.CreateCommand()
    $cmd.CommandTimeout = 0
    $cmd.CommandText = $Sql
    [void]$cmd.ExecuteNonQuery()
}

try {
    $conn.Open()

    Write-Host "Freezing SQL database writes..."
    Invoke-SqlNonQuery "ALTER DATABASE [$DbSafe] SET SUSPEND_FOR_SNAPSHOT_BACKUP = ON;"

    Write-Host "Taking ZFS snapshot on Proxmox..."
    ssh "$PveUser@$PveHost" "zfs snapshot $ZfsSnapshotArgs && zfs hold sqlsnap $ZfsSnapshotArgs"

    if ($LASTEXITCODE -ne 0) {
        throw "ZFS snapshot failed on $PveHost"
    }

    Write-Host "Writing SQL metadata backup..."

    Invoke-SqlNonQuery @"
BACKUP DATABASE [$DbSafe]
TO DISK = N'$BackupFileSql'
WITH METADATA_ONLY,
     MEDIADESCRIPTION = N'$MediaSql',
     NAME = N'$SnapName';
"@

    Write-Host "Snapshot backup completed:"
    Write-Host "  Snapshot: $ZfsSnapshotArgs"
    Write-Host "  Metadata: $BackupFile"
}
catch {
    Write-Warning $_

    try {
        Write-Warning "Attempting to unfreeze SQL database..."
        Invoke-SqlNonQuery "ALTER DATABASE [$DbSafe] SET SUSPEND_FOR_SNAPSHOT_BACKUP = OFF;"
    }
    catch {
        Write-Warning "Could not unfreeze cleanly. Check SQL Server error log."
    }

    throw
}
finally {
    $conn.Close()
}

Restore process

Here is how the restore process works:

• We connect to the corresponding SQL Server instance.
• We take the database offline.
• The volume dedicated to the StackOverflow database is taken offline.
• We connect to the hypervisor through SSH.
• We roll back the corresponding snapshot.
• We restore the database using the corresponding backup, which was created at the same time as the snapshot.

Powershell implementation (restore)

Here is the code used to perform the restore:

param(
    [string]$SqlInstance = "VM-WS25-SQL2",
    [string]$Database    = "StackOverflow",
    [string]$BackupFile  = "D:\Backups\StackOverflow_20260514T122642.bkm",
    [string]$SnapName    = "sql_StackOverflow_20260514T122642",
    [string]$PveHost     = "192.168.1.110",
    [string]$PveUser     = "MyUser",
    [string[]]$Zvols     = @("sqlpool/pve/vm-302-disk-0"),
    [string[]]$DatabaseDriveLetters = @("T"),
    [switch]$NoRecovery
)

$ErrorActionPreference = "Stop"

function Assert-SafeName {
    param(
        [string]$Value,
        [string]$Name,
        [string]$Pattern
    )

    if ($Value -notmatch $Pattern) {
        throw "$Name contained not allowed characters : $Value"
    }
}

function Normalize-DriveLetter {
    param([string]$DriveLetter)

    $letter = $DriveLetter.Trim().TrimEnd(":").ToUpperInvariant()

    if ($letter -notmatch '^[A-Z]$') {
        throw "Drive letter invalid : $DriveLetter"
    }

    return $letter
}

function Get-DiskForDriveLetter {
    param([string]$DriveLetter)

    $letter = Normalize-DriveLetter $DriveLetter

    $partition = Get-Partition -DriveLetter $letter -ErrorAction Stop
    $disk = $partition | Get-Disk -ErrorAction Stop

    return [pscustomobject]@{
        DriveLetter = $letter
        DiskNumber  = [int]$disk.Number
        IsOffline   = [bool]$disk.IsOffline
        FriendlyName = $disk.FriendlyName
        Size        = $disk.Size
    }
}

function Invoke-SshChecked {
    param([string]$Command)

    Write-Host "SSH $PveUser@$PveHost :: $Command"

    & ssh "$PveUser@$PveHost" "$Command"

    if ($LASTEXITCODE -ne 0) {
        throw "SSH command failed with code $LASTEXITCODE : $Command"
    }
}

function New-SqlConnection {
    $connString = "Server=$SqlInstance;Database=master;Integrated Security=True;TrustServerCertificate=True;Application Name=ZFS-TSQL-Restore-NoVmRestart;"
    return New-Object System.Data.SqlClient.SqlConnection $connString
}

function Invoke-SqlNonQuery {
    param([string]$Sql)

    $conn = New-SqlConnection

    try {
        $conn.Open()
        $cmd = $conn.CreateCommand()
        $cmd.CommandTimeout = 0
        $cmd.CommandText = $Sql
        [void]$cmd.ExecuteNonQuery()
    }
    finally {
        $conn.Close()
    }
}

function Invoke-SqlScalar {
    param([string]$Sql)

    $conn = New-SqlConnection

    try {
        $conn.Open()
        $cmd = $conn.CreateCommand()
        $cmd.CommandTimeout = 0
        $cmd.CommandText = $Sql
        return $cmd.ExecuteScalar()
    }
    finally {
        $conn.Close()
    }
}

function Set-DatabaseDisksOffline {
    param([object[]]$DiskInfos)

    $offlinedByScript = @()

    foreach ($diskInfo in ($DiskInfos | Sort-Object DiskNumber -Unique)) {
        if ($diskInfo.IsOffline) {
            Write-Host "Disque $($diskInfo.DiskNumber) déjà offline. Lecteur $($diskInfo.DriveLetter):"
            continue
        }

        Write-Host "Taking the Windows disk offline $($diskInfo.DiskNumber), drive $($diskInfo.DriveLetter):"
        Set-Disk -Number $diskInfo.DiskNumber -IsOffline $true

        $offlinedByScript += $diskInfo
    }

    return $offlinedByScript
}

function Set-DatabaseDisksOnline {
    param([object[]]$DiskInfos)

    foreach ($diskInfo in ($DiskInfos | Sort-Object DiskNumber -Unique)) {
        Write-Host "Bringing the Windows disk back online. $($diskInfo.DiskNumber), drive $($diskInfo.DriveLetter):"
        Set-Disk -Number $diskInfo.DiskNumber -IsOffline $false
    }

    Write-Host "Update-HostStorageCache..."
    Update-HostStorageCache
}

Assert-SafeName -Value $SnapName -Name "SnapName" -Pattern '^[A-Za-z0-9_.:-]{1,160}$'

foreach ($zvol in $Zvols) {
    Assert-SafeName -Value $zvol -Name "Zvol" -Pattern '^[A-Za-z0-9_.:/-]{1,240}$'
}

$DbQuoted = "[" + $Database.Replace("]", "]]") + "]"
$DbLiteral = $Database.Replace("'", "''")
$BackupFileSql = $BackupFile.Replace("'", "''")

$ZfsSnapshots = $Zvols | ForEach-Object { "$_@$SnapName" }
$ZfsSnapshotArgs = ($ZfsSnapshots | ForEach-Object { "'$_'" }) -join " "

$RecoveryOption = if ($NoRecovery) { "NORECOVERY" } else { "RECOVERY" }

$DatabaseDiskInfos = @()
$DisksOfflinedByScript = @()

Write-Host ""
Write-Host "Restore SQL Server from a ZFS snapshot, without restarting the VM"
Write-Host "SQL Instance : $SqlInstance"
Write-Host "Database     : $Database"
Write-Host "BackupFile   : $BackupFile"
Write-Host "DB volumes   : $($DatabaseDriveLetters -join ', ')"
Write-Host "Snapshots    :"
$ZfsSnapshots | ForEach-Object { Write-Host "  $_" }
Write-Host ""

try {
    Write-Host "Checking ZFS snapshots..."
    Invoke-SshChecked "zfs list -H -t snapshot -o name $ZfsSnapshotArgs >/dev/null"

    Write-Host "Identifying Windows disks containing SQL Server files..."
    foreach ($driveLetter in $DatabaseDriveLetters) {
        $diskInfo = Get-DiskForDriveLetter $driveLetter
        $DatabaseDiskInfos += $diskInfo

        Write-Host "Drive $($diskInfo.DriveLetter): -> Windows disk $($diskInfo.DiskNumber) [$($diskInfo.FriendlyName)]"
    }

    $backupDrive = $null
    if ($BackupFile -match '^([A-Za-z]):\\') {
        $backupDrive = Normalize-DriveLetter $Matches[1]

        try {
            $backupDiskInfo = Get-DiskForDriveLetter $backupDrive
            $targetDiskNumbers = @($DatabaseDiskInfos | ForEach-Object { $_.DiskNumber } | Select-Object -Unique)

            if ($targetDiskNumbers -contains $backupDiskInfo.DiskNumber) {
                throw @"
The backup file $BackupFile is located on drive $backupDrive, which is on the same Windows disk as the SQL Server data volume.
Taking the data disk offline would make the .bkm file inaccessible, and a rollback could also make the .bkm file disappear.
Move the .bkm file to C:, a network share, or another disk that is not rolled back.
"@
            }
        }
        catch {
            throw
        }
    }

    Write-Host "Checking whether the SQL Server database exists..."
    $DbExists = Invoke-SqlScalar "SELECT CASE WHEN DB_ID(N'$DbLiteral') IS NULL THEN 0 ELSE 1 END;"

    if ($DbExists -eq 1) {
        Write-Host "Taking database $Database OFFLINE..."
        Invoke-SqlNonQuery @"
ALTER DATABASE $DbQuoted SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
ALTER DATABASE $DbQuoted SET OFFLINE WITH ROLLBACK IMMEDIATE;
"@
    }
    else {
        Write-Host "Database $Database does not exist in SQL Server. Continuing with disk offline and ZFS rollback."
    }

    Write-Host "Taking Windows disks containing MDF/LDF files offline..."
    $DisksOfflinedByScript = Set-DatabaseDisksOffline -DiskInfos $DatabaseDiskInfos

    Write-Host "Rolling back ZFS snapshot..."
    $RollbackCommands = ($ZfsSnapshots | ForEach-Object { "zfs rollback -r '$_'" }) -join "; "
    Invoke-SshChecked "set -e; $RollbackCommands"

    Write-Host "Bringing Windows disks back online..."
    Set-DatabaseDisksOnline -DiskInfos $DisksOfflinedByScript
    $DisksOfflinedByScript = @()

    Write-Host "Short pause to let Windows and SQL Server detect the restored disk state..."
    Start-Sleep -Seconds 5

    Write-Host "Restoring SQL Server metadata-only backup..."

    $RestoreSql = @"
RESTORE DATABASE $DbQuoted
FROM DISK = N'$BackupFileSql'
WITH METADATA_ONLY,
     REPLACE,
     $RecoveryOption;
"@

    Invoke-SqlNonQuery $RestoreSql

    if (-not $NoRecovery) {
        Write-Host "Setting database back to MULTI_USER..."
        Invoke-SqlNonQuery @"
ALTER DATABASE $DbQuoted SET MULTI_USER;
"@
    }

    Write-Host ""
    Write-Host "Restore completed."
    Write-Host "Database : $Database"
    Write-Host "Snapshot : $SnapName"
    Write-Host "Backup   : $BackupFile"
}
catch {
    Write-Warning "Restore failed: $_"

    if ($DisksOfflinedByScript.Count -gt 0) {
        try {
            Write-Warning "Attempting to bring disks offlined by the script back online..."
            Set-DatabaseDisksOnline -DiskInfos $DisksOfflinedByScript
            $DisksOfflinedByScript = @()
        }
        catch {
            Write-Warning "Unable to automatically bring the disks back online. Check with Get-Disk."
        }
    }

    try {
        $DbExistsAfterError = Invoke-SqlScalar "SELECT CASE WHEN DB_ID(N'$DbLiteral') IS NULL THEN 0 ELSE 1 END;"

        if ($DbExistsAfterError -eq 1 -and -not $NoRecovery) {
            Write-Warning "Attempting to set the database back ONLINE/MULTI_USER..."
            Invoke-SqlNonQuery @"
ALTER DATABASE $DbQuoted SET ONLINE;
ALTER DATABASE $DbQuoted SET MULTI_USER;
"@
        }
    }
    catch {
        Write-Warning "Unable to automatically set the database back ONLINE/MULTI_USER."
    }

    throw
}

What does it look like?

We start the backup process:

We verify that the snapshot is present:

We verify that the backup is present:

We drop the StackOverflow database:

We start the restore process:

The database is available again. The restore took only a few seconds for a database of approximately 200 GB.

Major drawbacks

In my case, the solution is executed from the SQL Server itself. Ideally, it should rather be hosted on another server or client machine. We could also imagine running these scripts from a scheduler such as RedDeck, for example.

During the database restore, the database is switched to SINGLE_USER mode. This could be an issue if the applications using the database reconnect very frequently. A better approach would probably be to explicitly terminate the active sessions using the KILL command.

We have also not yet covered the use of a REST API.

Thank you. Amine Haloui

L’article SQL Server Snapshot Backup and Restore with Proxmox ZFS – Powershell implementation (2/4) est apparu en premier sur dbi Blog.