We are entering now to the infrastructure as code world and provisioning a SQL Server infrastructure is not excluded from the equation. This is especially true when it comes the Cloud regardless we are using IaaS or PaaS.
One great tool to use in such scenario is certainly terraform and I introduced it during the last PowerSaturday pre-conference named “DBA modern competencies”. Installation paradigms in the cloud differ from what we usually do on-premises either we are GUI-oriented or scripting-enthusiastic DBAs. In on-premises scenarios, building and delivering a software require both to deal often with a lot of hardware including servers, racks, network, cooling and so on. So, it makes sense to have one team for managing the hardware stuff (Ops) and another one to develop software (Devs). But nowadays, a shift is taking place and instead of managing their own data centers for some of their system components, many customers are moving to the cloud, taking advantage of services such as Azure, AWS and Google Cloud. This is a least what I begin to notice with some of my customers for some times now including some of their database environments. Instead of investing heavily in hardware, many Ops teams are spending all their time working on software, using tools such as Chef, Puppet, Terraform or Docker. In other words, instead of racking servers and plugging in network cables, many sysadmins are writing code. From a database administrator standpoint, I believe this is a good news because it will remove this boring part of the work without any real adding values.
The question that may rise probably is why to use Terraform (or equivalent) rather than a script? I mean script as ad-hoc script here … Personally, I use PowerShell based ah-hoc scripts to install and configure SQL Server for a while now at customer shops and PowerShell is part of IAC (Infrastructure As Code) tooling category. But using ad-hoc scripts come with some drawbacks. First of all, using a programming language like PowerShell implies you have to write completely custom code for every task and it is not a big deal as long as you manage few components in the infrastructure. But what if you’re dealing with hundred or thousand of servers including databases, network, load balancers and so on? Do you really want to maintain a big and unmaintainable ad-hoc script repository especially if you work in a collaborative way? In this case you have to rely on a tool designed for such job. Furthermore, writing an ad hoc script that works once isn’t often too difficult but writing one that works correctly even if you run it over and over again is a lot harder and impotent code is an important part of modern infrastructure.
In my case, we also investigated on different configuration management tools like Ansible but they are mostly designed to install and manage software on existing servers. In the context of the cloud, in most cases servers do not exist and you have to provide one (or many) including the related infrastructure like virtual network, virtual IP, virtual machine, disks and so forth … So here comes the big interest of using Terraform in such scenario. Terraform is an Hashicop product and comes with open source vs Enterprise version. I’m using the open source one with 01.2 version at the moment of writing this blog post as shown below:
$ terraform version Terraform v0.12.0 + provider.azurerm v1.31.0 + provider.template v2.1.2
In my case, it may be very helpful to provision servers on Azure. Microsoft already invested a lot to introduce Terraform as provisioning tool for different Azure services including SQL Azure DBs with azurerm_sql_database or azurerm_sql_server providers. But the story is not the same when it comes SQL Server virtual machines on Azure. The only provider available to provision an Azure VM is azurerm_virtual_machine but it doesn’t include the SQL Server configuration part. In this blog post I want to expose some challenges I faced to make it possible and you probably want to be aware of 🙂
Let’s say that in fact, we have to rely on the azurerm_template_deployment provider in this case but at the cost of some complexity. Indeed, you have to introduce the SQL Server template Microsoft.SqlVirtualMachine/SqlVirtualMachines to the Terraform files with all the difficulties that is implies. First, from my opinion adding a SQL Server template file leads to additional complexity that is at the opposite of what we expect from Terraform. We lose some code simplification and maintainability somewhere and this is by far my main disappointment. Anyway, in addition let’s say that debugging a Terraform deployment with Azure templates may be time-consuming task especially if you have to deal with syntax error issues. You often have to take a look at the Azure event logs directly to know what is happening. Second, the parameters are all string type and we need to convert them to the right type with Azure template variables. Finally, as far as I know, it is not possible to use arrays as input parameters within the Azure template and refactoring from the initial server template is requested accordingly. This is the case of the SQL Server dataDisks configuration parameter for instance. I don’t know if Microsoft is planning something on this topic but my expectation is to get something more “Terraform-integrated” in the future.
Let’s finish this blog post with my Terraform files used in my SQL Server VM Azure provisioning workflow that is as follows:
Create a resource group => Create a virtual network + subnet => Create SQL Server Azure VM with custom configuration
- The main configuration file. The resource section is understanble and generally speaking the configuration file remains maintenable enough here with the declarative / descriptive way to provision resources.
# =============== VARIABLES =============== #
variable "prefix" {
type = string
default = "dbi"
}
variable "resourcegroup" {
type = string
}
variable "location" {
type = string
default = "westeurope"
}
variable "subscriptionId" {
type = string
}
variable "virtualmachinename" {
type = string
}
variable "virtualMachineSize" {
type = string
}
variable "adminUsername" {
type = string
}
variable "adminUserPassword" {
type = string
}
variable "image_ref_offer" {
type = string
}
variable "image_ref_sku" {
type = string
}
variable "image_ref_version" {
type = string
}
variable "osDiskType" {
type = string
default = "Premium_LRS"
}
variable "sqlVirtualMachineLocation" {
type = string
default = "westeurope"
}
variable "sqlServerLicenseType" {
type = string
}
variable "sqlPortNumber" {
type = string
default = "1433"
}
variable "sqlStorageDisksCount" {
type = string
default = "1"
}
variable "diskSqlSizeGB" {
type = string
default = "1024"
}
variable "sqlDisklType" {
type = string
default = "Premium_LRS"
}
variable "sqlStorageWorkloadType" {
type = string
default = "GENERAL"
}
variable "sqlAuthenticationLogin" {
type = string
}
variable "sqlAuthenticationPassword" {
type = string
}
variable "sqlConnectivityType" {
type = string
}
variable "sqlAutopatchingDayOfWeek" {
type = string
default = "Sunday"
}
variable "sqlAutopatchingStartHour" {
type = string
default = "2"
}
variable "sqlAutopatchingWindowDuration" {
type = string
default = "60"
}
variable "diagnosticsStorageAccountName" {
type = string
}
variable "tag" {
type = string
}
# =============== TEMPLATES =============== #
data "template_file" "sqlvm" {
template = file("./Templates/sql_vm_azure_dbi.json")
}
# =============== RESOURCES =============== #
resource "azurerm_resource_group" "sqlvm" {
name = var.resourcegroup
location = "West Europe"
}
resource "azurerm_virtual_network" "sqlvm" {
name = "${var.resourcegroup}-vnet"
address_space = ["172.20.0.0/24"]
location = azurerm_resource_group.sqlvm.location
resource_group_name = azurerm_resource_group.sqlvm.name
}
resource "azurerm_subnet" "internal" {
name = "default"
resource_group_name = var.resourcegroup
virtual_network_name = azurerm_virtual_network.sqlvm.name
address_prefix = "172.20.0.0/24"
}
resource "azurerm_template_deployment" "sqlvm" {
name = "${var.prefix}-template"
resource_group_name = azurerm_resource_group.sqlvm.name
template_body = data.template_file.sqlvm.rendered
#DEPLOY
# =============== PARAMETERS =============== #
parameters = {
"location" = var.location # Location (westeurope by default)
"networkInterfaceName" = "${var.prefix}-${var.virtualmachinename}-interface" # Virtual machine interace name
"enableAcceleratedNetworking" = "true" # Enable Accelerating networking (always YES)
"networkSecurityGroupName" = "${var.prefix}-${var.virtualmachinename}-nsg" # NSG name (computed)
"subnetName" = azurerm_subnet.internal.name # Resource subnet
"virtualNetworkId" = "/subscriptions/${var.subscriptionId}/resourceGroups/${var.resourcegroup}/providers/Microsoft.Network/virtualNetworks/${var.resourcegroup}-vnet"
"publicIpAddressName" = "${var.prefix}-${var.virtualmachinename}-ip" # Public IP Address name (computed)
"publicIpAddressType" = "Dynamic" # Public IP allocation (Dynamic, Static)
"publicIpAddressSku" = "Basic" # Public IP Address sku (None, Basic, Advanced)
"virtualMachineName" = "${var.prefix}-${var.virtualmachinename}" # Virtual machine name (computed)
"virtualMachineRG" = var.resourcegroup # Resource group for resources
"virtualMachineSize" = var.virtualMachineSize # Virtual machine size (Standard_DS13_v2)
"image_ref_offer" = var.image_ref_offer # SQL Server Image Offer (SQL2017-WS2016, ...)
"image_ref_sku" = var.image_ref_sku # SQL Server Image SKU (SQLDEV, ...)
"image_ref_version" = var.image_ref_version # SQL Server Image version (latest, <version number>)
"adminUsername" = var.adminUsername # Virtual machine user name
"adminUserPassword" = var.adminUserPassword # Virtual machine user password
"osDiskType" = var.osDiskType # OS Disk type (Premium_LRS by default)
"sqlDisklType" = var.sqlDisklType # SQL Disk type Premium_LRS by default)
"diskSqlSizeGB" = var.diskSqlSizeGB # SQL Disk size (GB)
"diagnosticsStorageAccountName" = var.diagnosticsStorageAccountName # Diagnostics info - storage account name
"diagnosticsStorageAccountId" = "/subscriptions/${var.subscriptionId}/resourceGroups/${var.resourcegroup}/providers/Microsoft.Storage/storageAccounts/${var.diagnosticsStorageAccountName}" # Storage account must exist
"diagnosticsStorageAccountType" = "Standard_LRS" # Diagnostics info - storage account type
"diagnosticsStorageAccountKind" = "Storage" # Diagnostics info - storage type
"sqlVirtualMachineLocation" = var.sqlVirtualMachineLocation # Virtual machine location
"sqlVirtualMachineName" = "${var.prefix}-${var.virtualmachinename}" # Virtual machine name
"sqlServerLicenseType" = var.sqlServerLicenseType # SQL Server license type. - PAYG or AHUB
"sqlConnectivityType" = var.sqlConnectivityType # LOCAL, PRIVATE, PUBLIC
"sqlPortNumber" = var.sqlPortNumber # SQL listen port
"sqlStorageDisksCount" = var.sqlStorageDisksCount # Nb of SQL disks to provision
"sqlStorageWorkloadType" = var.sqlStorageWorkloadType # Workload type GENERAL, OLTP, DW
"sqlStorageDisksConfigurationType" = "NEW" # Configuration type NEW
"sqlStorageStartingDeviceId" = "2" # Storage starting device id => Always 2
"sqlStorageDeploymentToken" = "8528" # Deployment Token
"sqlAutopatchingDayOfWeek" = var.sqlAutopatchingDayOfWeek # Day of week to apply the patch on. - Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday
"sqlAutopatchingStartHour" = var.sqlAutopatchingStartHour # Hour of the day when patching is initiated. Local VM time
"sqlAutopatchingWindowDuration" = var.sqlAutopatchingWindowDuration # Duration of patching
"sqlAuthenticationLogin" = var.sqlAuthenticationLogin # Login SQL
"sqlAuthUpdatePassword" = var.sqlAuthenticationPassword # Login SQL Password
"rServicesEnabled" = "false" # No need to enable R services
"tag" = var.tag # Resource tags
}
deployment_mode = "Incremental" # Deployment => incremental (complete is too destructive in our case)
}
- The SQL Server template from customized from SqlVirtualMachine/SqlVirtualMachine template. As said previously, the hardest part of the deployment. Hope to see it removed in the future!
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "string"
},
"networkInterfaceName": {
"type": "string"
},
"enableAcceleratedNetworking": {
"type": "string"
},
"networkSecurityGroupName": {
"type": "string"
},
"subnetName": {
"type": "string"
},
"virtualNetworkId": {
"type": "string"
},
"publicIpAddressName": {
"type": "string"
},
"publicIpAddressType": {
"type": "string"
},
"publicIpAddressSku": {
"type": "string"
},
"virtualMachineName": {
"type": "string"
},
"virtualMachineRG": {
"type": "string"
},
"osDiskType": {
"type": "string"
},
"virtualMachineSize": {
"type": "string"
},
"image_ref_offer": {
"type": "string"
},
"image_ref_sku": {
"type": "string"
},
"image_ref_version": {
"type": "string"
},
"adminUsername": {
"type": "string"
},
"adminUserPassword": {
"type": "string"
},
"diagnosticsStorageAccountName": {
"type": "string"
},
"diagnosticsStorageAccountId": {
"type": "string"
},
"diagnosticsStorageAccountType": {
"type": "string"
},
"diagnosticsStorageAccountKind": {
"type": "string"
},
"sqlVirtualMachineLocation": {
"type": "string"
},
"sqlVirtualMachineName": {
"type": "string"
},
"sqlServerLicenseType": {
"type": "string"
},
"sqlConnectivityType": {
"type": "string"
},
"sqlPortNumber": {
"type": "string"
},
"sqlStorageDisksCount": {
"type": "string"
},
"sqlDisklType": {
"type": "string"
},
"diskSqlSizeGB": {
"type": "string"
},
"sqlStorageWorkloadType": {
"type": "string"
},
"sqlStorageDisksConfigurationType": {
"type": "string"
},
"sqlStorageStartingDeviceId": {
"type": "string"
},
"sqlStorageDeploymentToken": {
"type": "string"
},
"sqlAutopatchingDayOfWeek": {
"type": "string"
},
"sqlAutopatchingStartHour": {
"type": "string"
},
"sqlAutopatchingWindowDuration": {
"type": "string"
},
"sqlAuthenticationLogin": {
"type": "string"
},
"sqlAuthUpdatePassword": {
"type": "string"
},
"rServicesEnabled": {
"type": "string"
},
"tag": {
"type": "string"
}
},
"variables": {
"nsgId": "[resourceId(resourceGroup().name, 'Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]",
"vnetId": "[parameters('virtualNetworkId')]",
"subnetRef": "[concat(variables('vnetId'), '/subnets/', parameters('subnetName'))]",
"dataDisks": [
{
"lun": "0",
"createOption": "empty",
"caching": "ReadOnly",
"writeAcceleratorEnabled": false,
"id": null,
"name": null,
"storageAccountType": "[parameters('sqlDisklType')]",
"diskSizeGB": "[int(parameters('diskSqlSizeGB'))]"
}
]
},
"resources": [
{
"name": "[parameters('networkInterfaceName')]",
"type": "Microsoft.Network/networkInterfaces",
"apiVersion": "2018-10-01",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Network/networkSecurityGroups/', parameters('networkSecurityGroupName'))]",
"[concat('Microsoft.Network/publicIpAddresses/', parameters('publicIpAddressName'))]"
],
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"subnet": {
"id": "[variables('subnetRef')]"
},
"privateIPAllocationMethod": "Dynamic",
"publicIpAddress": {
"id": "[resourceId(resourceGroup().name, 'Microsoft.Network/publicIpAddresses', parameters('publicIpAddressName'))]"
}
}
}
],
"enableAcceleratedNetworking": "[parameters('enableAcceleratedNetworking')]",
"networkSecurityGroup": {
"id": "[variables('nsgId')]"
}
},
"tags": {
"Environment": "[parameters('tag')]"
}
},
{
"name": "[parameters('networkSecurityGroupName')]",
"type": "Microsoft.Network/networkSecurityGroups",
"apiVersion": "2019-02-01",
"location": "[parameters('location')]",
"properties": {
"securityRules": [
{
"name": "RDP",
"properties": {
"priority": 300,
"protocol": "TCP",
"access": "Allow",
"direction": "Inbound",
"sourceAddressPrefix": "*",
"sourcePortRange": "*",
"destinationAddressPrefix": "*",
"destinationPortRange": "3389"
}
}
]
},
"tags": {
"Environment": "[parameters('tag')]"
}
},
{
"name": "[parameters('publicIpAddressName')]",
"type": "Microsoft.Network/publicIpAddresses",
"apiVersion": "2019-02-01",
"location": "[parameters('location')]",
"properties": {
"publicIpAllocationMethod": "[parameters('publicIpAddressType')]"
},
"sku": {
"name": "[parameters('publicIpAddressSku')]"
},
"tags": {
"Environment": "[parameters('tag')]"
}
},
{
"name": "[parameters('virtualMachineName')]",
"type": "Microsoft.Compute/virtualMachines",
"apiVersion": "2018-10-01",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Network/networkInterfaces/', parameters('networkInterfaceName'))]",
"[concat('Microsoft.Storage/storageAccounts/', parameters('diagnosticsStorageAccountName'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "[parameters('virtualMachineSize')]"
},
"storageProfile": {
"osDisk": {
"createOption": "fromImage",
"managedDisk": {
"storageAccountType": "[parameters('osDiskType')]"
}
},
"imageReference": {
"publisher": "MicrosoftSQLServer",
"offer": "[parameters('image_ref_offer')]",
"sku": "[parameters('image_ref_sku')]",
"version": "[parameters('image_ref_version')]"
},
"copy": [
{
"name": "dataDisks",
"count": "[length(variables('dataDisks'))]",
"input": {
"lun": "[variables('dataDisks')[copyIndex('dataDisks')].lun]",
"createOption": "[variables('dataDisks')[copyIndex('dataDisks')].createOption]",
"caching": "[variables('dataDisks')[copyIndex('dataDisks')].caching]",
"writeAcceleratorEnabled": "[variables('dataDisks')[copyIndex('dataDisks')].writeAcceleratorEnabled]",
"diskSizeGB": "[variables('dataDisks')[copyIndex('dataDisks')].diskSizeGB]",
"managedDisk": {
"id": "[coalesce(variables('dataDisks')[copyIndex('dataDisks')].id, if(equals(variables('dataDisks')[copyIndex('dataDisks')].name, json('null')), json('null'), resourceId('Microsoft.Compute/disks', variables('dataDisks')[copyIndex('dataDisks')].name)))]",
"storageAccountType": "[variables('dataDisks')[copyIndex('dataDisks')].storageAccountType]"
}
}
}
]
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName'))]"
}
]
},
"osProfile": {
"computerName": "[parameters('virtualMachineName')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminUserPassword')]",
"windowsConfiguration": {
"enableAutomaticUpdates": true,
"provisionVmAgent": true
}
},
"licenseType": "Windows_Server",
"diagnosticsProfile": {
"bootDiagnostics": {
"enabled": true,
"storageUri": "[concat('https://', parameters('diagnosticsStorageAccountName'), '.blob.core.windows.net/')]"
}
}
},
"tags": {
"Environment": "[parameters('tag')]"
}
},
{
"name": "[parameters('diagnosticsStorageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2018-07-01",
"location": "[parameters('location')]",
"properties": {},
"sku": {
"name": "[parameters('diagnosticsStorageAccountType')]"
},
"tags": {
"Environment": "[parameters('tag')]"
}
},
{
"name": "[parameters('sqlVirtualMachineName')]",
"type": "Microsoft.SqlVirtualMachine/SqlVirtualMachines",
"apiVersion": "2017-03-01-preview",
"location": "[parameters('sqlVirtualMachineLocation')]",
"properties": {
"virtualMachineResourceId": "[resourceId('Microsoft.Compute/virtualMachines', parameters('sqlVirtualMachineName'))]",
"sqlServerLicenseType": "[parameters('sqlServerLicenseType')]",
"AutoPatchingSettings": {
"Enable": true,
"DayOfWeek": "[parameters('sqlAutopatchingDayOfWeek')]",
"MaintenanceWindowStartingHour": "[parameters('sqlAutopatchingStartHour')]",
"MaintenanceWindowDuration": "[parameters('sqlAutopatchingWindowDuration')]"
},
"KeyVaultCredentialSettings": {
"Enable": false,
"CredentialName": ""
},
"ServerConfigurationsManagementSettings": {
"SQLConnectivityUpdateSettings": {
"ConnectivityType": "[parameters('sqlConnectivityType')]",
"Port": "[parameters('sqlPortNumber')]",
"SQLAuthUpdateUserName": "[parameters('sqlAuthenticationLogin')]",
"SQLAuthUpdatePassword": "[parameters('sqlAuthUpdatePassword')]"
},
"SQLWorkloadTypeUpdateSettings": {
"SQLWorkloadType": "[parameters('sqlStorageWorkloadType')]"
},
"SQLStorageUpdateSettings": {
"DiskCount": "[parameters('sqlStorageDisksCount')]",
"DiskConfigurationType": "[parameters('sqlStorageDisksConfigurationType')]",
"StartingDeviceID": "[parameters('sqlStorageStartingDeviceId')]"
},
"AdditionalFeaturesServerConfigurations": {
"IsRServicesEnabled": "[parameters('rServicesEnabled')]"
}
}
},
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachines', parameters('sqlVirtualMachineName'))]"
],
"tags": {
"Environment": "[parameters('tag')]"
}
}
],
"outputs": {
"adminUsername": {
"type": "string",
"value": "[parameters('adminUsername')]"
}
}
}
- The terraform.tfvars file. This file (with sensible parameters) should never checked into source control obviously
# VM configuration subscriptionId="xxxxxxx" resourcegroup="external-rg" virtualMachineSize="Standard_DS13_v2" virtualmachinename="sql1" adminUsername="clustadmin" adminUserPassword="xxxxxx" diagnosticsStorageAccountName="dab" tag="SQLDEV" # Image reference configuration image_ref_offer="SQL2017-WS2016" image_ref_sku="SQLDEV" image_ref_version="latest" # SQL configuration sqlServerLicenseType="PAYG" sqlAuthenticationLogin="sqladmin" sqlAuthenticationPassword="xxxxx" sqlConnectivityType="Public" diskSqlSizeGB="1024" sqlStorageWorkloadType="OLTP" sqlPortNumber="5040" sqlAutopatchingDayOfWeek="Sunday" sqlAutopatchingStartHour="2" sqlAutopatchingWindowDuration="60"
This is up to you to customize this template for your own purpose. Let’s deploy it:
$ terraform refresh --var-file=vm.tfvars data.template_file.sqlvm: Refreshing state... azurerm_template_deployment.sqlvm: Creating... azurerm_template_deployment.sqlvm: Still creating... [10s elapsed] azurerm_template_deployment.sqlvm: Still creating... [20s elapsed] azurerm_template_deployment.sqlvm: Still creating... [30s elapsed] azurerm_template_deployment.sqlvm: Still creating... [40s elapsed] ...
Few minutes afterwards, my SQL Server VM on Azure is provisioned
$ az resource list --tag Environment=SQLDEV --query "[].{resource:resourceGroup,name:name,location:location,Type:type}" --out table
Resource Name Location Type
----------- -------------------------------------------------- ---------- ----------------------------------------------
EXTERNAL-RG dbi-sql1_disk2_1cacce801795410fb406844cfb1f9317 westeurope Microsoft.Compute/disks
EXTERNAL-RG dbi-sql1_OsDisk_1_78558bdf9a9648f29d78cfbb36d9bed9 westeurope Microsoft.Compute/disks
external-rg dbi-sql1 westeurope Microsoft.Compute/virtualMachines
external-rg dbi-sql1-interface westeurope Microsoft.Network/networkInterfaces
external-rg dbi-sql1-nsg westeurope Microsoft.Network/networkSecurityGroups
external-rg dbi-sql1-ip westeurope Microsoft.Network/publicIPAddresses
external-rg dbi-sql1 westeurope Microsoft.SqlVirtualMachine/SqlVirtualMachines
external-rg dab westeurope Microsoft.Storage/storageAccounts
Happy deployment !!
By David Barbarin
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/12/microsoft-square.png)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ENB_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/10/JPC_wev-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2024/03/AHI_web.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2023/03/KKE_web-min-scaled.jpg)