{"id":11747,"date":"2018-10-04T06:27:29","date_gmt":"2018-10-04T04:27:29","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/"},"modified":"2018-10-04T06:27:29","modified_gmt":"2018-10-04T04:27:29","slug":"first-steps-into-sql-server-2019-availability-groups-on-k8s","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/","title":{"rendered":"First steps into SQL Server 2019 availability groups on K8s"},"content":{"rendered":"<p>A couple of weeks ago, Microsoft announced the <a href=\"https:\/\/cloudblogs.microsoft.com\/sqlserver\/2018\/09\/24\/sql-server-2019-preview-combines-sql-server-and-apache-spark-to-create-a-unified-data-platform\/\" target=\"_blank\" rel=\"noopener noreferrer\">first public CTP version of next SQL Server version\u00a0(CTP2)<\/a>. It is not a surprise, the SQL Server vNext becomes SQL Server 2019 and there are a plenty of enhancements as well as new features to discover. But for now, let\u2019s start with likely one of my favorites: availability groups on Kurbernetes (aka K8s). As far I may see from customers and hear from my colleagues as well, we assist to a strong adoption of K8s with OpenShift as a main driver. I would not be surprised to see some SQL Server pods at customer shops in a near future, especially with the support of availability groups on K8s. From my opinion, that is definitely something that was missing in the previous for microservices architectures or not, for either quality or production environments.<\/p>\n<p><a href=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-0-AG-K8s.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-28212\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-0-AG-K8s.jpg\" alt=\"blog 143 - 0 - AG K8s\" width=\"413\" height=\"273\" \/><\/a><\/p>\n<p>Well, I decided to learn more about this new feature but let\u2019s say this write-up concerns the CTP 2.0 version and chances are things will likely change in the future. So, don\u2019t focus strictly on my words or commands I\u2019m using in this blog post.<\/p>\n<p>It is some time since I used the Service Azure Kubernetes (AKS) and I already wrote about it in a <a href=\"https:\/\/www.dbi-services.com\/blog\/introducing-sql-server-on-kubernetes\/\" target=\"_blank\" rel=\"noopener noreferrer\">previous blog post<\/a>. I used the same environment to deploy my first availability group on K8s. It was definitely an interesting experience because it involved getting technical skills about K8s infrastructure.<\/p>\n<p>So, let\u2019s set briefly the context with my K8s cluster on Azure that is composed of 3 agent nodes as shown below:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get nodes -o wide\nNAME                       STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME\naks-nodepool1-78763348-0   Ready     agent     126d      v1.9.6    &lt;none&gt;        Ubuntu 16.04.4 LTS   4.13.0-1016-azure   docker:\/\/1.13.1\naks-nodepool1-78763348-1   Ready     agent     126d      v1.9.6    &lt;none&gt;        Ubuntu 16.04.4 LTS   4.13.0-1016-azure   docker:\/\/1.13.1\naks-nodepool1-78763348-2   Ready     agent     35d       v1.9.6    &lt;none&gt;        Ubuntu 16.04.5 LTS   4.15.0-1023-azure   docker:\/\/1.13.1<\/pre>\n<p>&nbsp;<\/p>\n<p>I also used a custom namespace &#8211; agdev &#8211; to scope my availability group resources names.<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get ns\nNAME           STATUS        AGE\nag1            Terminating   23h\nagdev          Active        10h\nazure-system   Active        124d\ndefault        Active        124d\nkube-public    Active        124d\nkube-system    Active        124d<\/pre>\n<p>&nbsp;<\/p>\n<p>Referring to the <a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/linux\/sql-server-ag-kubernetes?view=sqlallproducts-allversions\" target=\"_blank\" rel=\"noopener noreferrer\">Microsoft documentation<\/a>, the SQL secrets (including master key and SA password secrets) are ready for use:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get secret sql-secrets -n agdev\nNAME                   TYPE                                  DATA      AGE\nsql-secrets            Opaque                                2         1d\n\n$ kubectl describe secret sql-secrets -n agdev\nName:         sql-secrets\nNamespace:    default\nLabels:       &lt;none&gt;\nAnnotations:  &lt;none&gt;\n\nType:  Opaque\n\nData\n====\nmasterkeypassword:  14 bytes\nsapassword:         14 bytes<\/pre>\n<p>&nbsp;<\/p>\n<ul>\n<li>The operator<\/li>\n<\/ul>\n<p>The first component to deploy is the operator which is a very important component in this infrastructure and that builds upon the basic Kubernetes resource and controller concepts. Kubernetes has a very pluggable way to add your own logic in the form of a controller in addition of existing built-in controllers as the old fashion replication controller, the replica sets and deployments. All of them are suitable for stateless applications but the story is not the same when we have to deal with stateful systems like databases because those system require specific application domain knowledge to correctly scale, upgrade and reconfigure while protecting against data loss or unavailability. For example, how to deal correctly with availability groups during a crash of pod? If we think about it, the work doesn\u2019t consist only in restarting the crashing pod but the system will also have to execute custom tasks in a background including electing of a new primary (aka leader election), ensuring a safe transition during the failover period to avoid split brain scenarios etc.<\/p>\n<p>Deploying the mssql-operator includes the creation of a new pod:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get pods -n agdev -l app=mssql-operator\nNAME                              READY     STATUS    RESTARTS   AGE\nmssql-operator-67447c4bd8-s6tbv   1\/1       Running   0          11h<\/pre>\n<p>&nbsp;<\/p>\n<p>Let\u2019s go further by getting more details about this pod:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl describe pod -n agdev mssql-operator-67447c4bd8-s6tbv\nName:           mssql-operator-67447c4bd8-s6tbv\nNamespace:      agdev\nNode:           aks-nodepool1-78763348-0\/10.240.0.4\nStart Time:     Mon, 01 Oct 2018 08:12:47 +0200\nLabels:         app=mssql-operator\n                pod-template-hash=2300370684\nAnnotations:    &lt;none&gt;\nStatus:         Running\nIP:             10.244.1.56\nControlled By:  ReplicaSet\/mssql-operator-67447c4bd8\nContainers:\n  mssql-operator:\n    Container ID:  docker:\/\/148ba4b8ccd91159fecc3087dd4c0b7eb7feb36be4b3b5124314121531cd3a3c\n    Image:         mcr.microsoft.com\/mssql\/ha:vNext-CTP2.0-ubuntu\n    Image ID:      docker-pullable:\/\/mcr.microsoft.com\/mssql\/ha@sha256:c5d20c8b34ea096a845de0222441304a14ad31a447d79904bafaf29f898704d0\n    Port:          &lt;none&gt;\n    Host Port:     &lt;none&gt;\n    Command:\n      \/mssql-server-k8s-operator\n    State:          Running\n      Started:      Mon, 01 Oct 2018 08:13:32 +0200\n    Ready:          True\n    Restart Count:  0\n    Environment:\n      MSSQL_K8S_NAMESPACE:  agdev (v1:metadata.namespace)\n    Mounts:\n      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from mssql-operator-token-bd5gc (ro)\n\u2026\nVolumes:\n  mssql-operator-token-bd5gc:\n    Type:        Secret (a volume populated by a Secret)\n    SecretName:  mssql-operator-token-bd5gc\n    Optional:    false<\/pre>\n<p>&nbsp;<\/p>\n<p>Some interesting items to note here:<\/p>\n<ul>\n<li>The SQL Server CTP image &#8211; mcr.microsoft.com\/mssql\/ha &#8211; comes from the new Microsoft Container Registry (MCR). The current tag is vNext-CTP2.0-ubuntu at the moment of this write-up<\/li>\n<li>Volume secret is mounted to pass sensitive data that concerns a K8s service account used by the pod. In fact, the deployment of availability groups implies the creation of multiple service accounts<\/li>\n<\/ul>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl describe secret -n agdev mssql-operator-token-bd5gc\nName:         mssql-operator-token-bd5gc\nNamespace:    agdev\nLabels:       &lt;none&gt;\nAnnotations:  kubernetes.io\/service-account.name=mssql-operator\n              kubernetes.io\/service-account.uid=03cb111e-c541-11e8-a34a-0a09b8f01b34\n\nType:  kubernetes.io\/service-account-token\n\nData\n====\nnamespace:  5 bytes\ntoken:      xxxx\nca.crt:     1720 bytes<\/pre>\n<p>&nbsp;<\/p>\n<p>The command is \/<strong>mssql-server-k8s-operator<\/strong> that is a binary file like other mssql-server* files packaged in the new SQL Server image and which are designed to respond to different events by appropriated actions like updating K8s resources:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl exec -ti -n agdev mssql-operator-67447c4bd8-s6tbv -- \/bin\/bash\nroot@mssql-operator-67447c4bd8-s6tbv:\/# ll mssql*\n-rwxrwxr-x 1 root root 32277998 Sep 19 16:00 mssql-server-k8s-ag-agent*\n-rwxrwxr-x 1 root root 31848041 Sep 19 16:00 mssql-server-k8s-ag-agent-supervisor*\n-rwxrwxr-x 1 root root 31336739 Sep 19 16:00 mssql-server-k8s-failover*\n-rwxrwxr-x 1 root root 32203064 Sep 19 16:00 mssql-server-k8s-health-agent*\n-rwxrwxr-x 1 root root 31683946 Sep 19 16:00 mssql-server-k8s-init-sql*\n-rwxrwxr-x 1 root root 31422517 Sep 19 16:00 mssql-server-k8s-operator*\n-rwxrwxr-x 1 root root 31645032 Sep 19 16:00 mssql-server-k8s-rotate-creds*\n\nroot@mssql-operator-67447c4bd8-s6tbv:\/# file mssql-server-k8s-operator\nmssql-server-k8s-operator: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter \/lib64\/ld-linux-x86-64.so.2, not stripped<\/pre>\n<p>&nbsp;<\/p>\n<ul>\n<li>The SQL Server instances and AGs<\/li>\n<\/ul>\n<p>The next step consisted in running the SQL Server AG deployment. Looking at the manifest file, we may notice we deploy custom SQL Server objects (<strong>kind: SqlServer<\/strong>) from new <strong><em>mssql.microsoft.com<\/em><\/strong> API installed previously as well as their corresponding services to expose SQL Server pods to the external traffic.<\/p>\n<p>The deployment includes 3 StatefulSets that manage pods with 2 containers, respectively the SQL Server engine and its agent (HA supervisor). I was surprised to not see a deployment with <strong>kind: StatefulSet<\/strong> but I got the confirmation that the \u201clogic\u201d is encapsulated in the SqlServer object definition. Why StatfulSets here? Well, because they are more valuable for applications like databases by providing, inter alia, stable and unique network identifiers as well as stable and persistent storage. Stateless pods do not provide such capabilities. To meet StafulSet prerequisites, we need first to define persistent volumes for each SQL Server pod. Recent version of K8s allows to use dynamic provisioning and this is exactly what is used in the initial Microsoft deployment file with the <strong>instanceRootVolumeClaimTemplate<\/strong>:<\/p>\n<pre class=\"brush: xml; gutter: true; first-line: 1\">instanceRootVolumeClaimTemplate:\n   accessModes: [ReadWriteOnce]\n   resources:\n     requests: {storage: 5Gi}\n   storageClass: default<\/pre>\n<p>&nbsp;<\/p>\n<p>However, in my context I already created persistent volumes for previous tests as shown below:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get pv -n agdev\nNAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                STORAGECLASS   REASON    AGE\npvc-cb299d79-c5b4-11e8-a34a-0a09b8f01b34   10Gi       RWO            Delete           Bound     agdev\/mssql-data-1   azure-disk               9h\npvc-cb4915b4-c5b4-11e8-a34a-0a09b8f01b34   10Gi       RWO            Delete           Bound     agdev\/mssql-data-2   azure-disk               9h\npvc-cb67cd06-c5b4-11e8-a34a-0a09b8f01b34   10Gi       RWO            Delete           Bound     agdev\/mssql-data-3   azure-disk               9h\n\n$ kubectl get pvc -n agdev\nNAME           STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\nmssql-data-1   Bound     pvc-cb299d79-c5b4-11e8-a34a-0a09b8f01b34   10Gi       RWO            azure-disk     9h\nmssql-data-2   Bound     pvc-cb4915b4-c5b4-11e8-a34a-0a09b8f01b34   10Gi       RWO            azure-disk     9h\nmssql-data-3   Bound     pvc-cb67cd06-c5b4-11e8-a34a-0a09b8f01b34   10Gi       RWO            azure-disk     9h<\/pre>\n<p>&nbsp;<\/p>\n<p>So, I changed a little bit the initial manifest file for each SqlServer object with my existing persistent claims:<\/p>\n<pre class=\"brush: xml; gutter: true; first-line: 1\">instanceRootVolume:\n    persistentVolumeClaim:\n      claimName: mssql-data-1<\/pre>\n<p>&#8230;<\/p>\n<pre class=\"brush: xml; gutter: true; first-line: 1\">instanceRootVolume:\n    persistentVolumeClaim:\n      claimName: mssql-data-2<\/pre>\n<p>&#8230;<\/p>\n<pre class=\"brush: xml; gutter: true; first-line: 1\">instanceRootVolume:\n    persistentVolumeClaim:\n      claimName: mssql-data-3<\/pre>\n<p>&nbsp;<\/p>\n<p>Furthermore, next prerequisite for StatefulSet consists in using a headless service and this exactly we may find with the creation of ag1 service during the deployment:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get svc -n agdev\nNAME          TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)             AGE\nag1           ClusterIP      None           &lt;none&gt;          1433\/TCP,5022\/TCP   1d<\/pre>\n<p>&nbsp;<\/p>\n<p>I also noticed some other interesting items like extra pods in completed state:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get pods -n agdev -l app!=mssql-operator\nNAME                            READY     STATUS      RESTARTS   AGE\nmssql-initialize-mssql1-plh8l   0\/1       Completed   0          9h\nmssql-initialize-mssql2-l6z8m   0\/1       Completed   0          9h\nmssql-initialize-mssql3-wrbkl   0\/1       Completed   0          9h\nmssql1-0                        2\/2       Running     0          9h\nmssql2-0                        2\/2       Running     0          9h\nmssql3-0                        2\/2       Running     0          9h\n\n$ kubectl get sts -n agdev\nNAME      DESIRED   CURRENT   AGE\nmssql1    1         1         9h\nmssql2    1         1         9h\nmssql3    1         1         9h<\/pre>\n<p>&nbsp;<\/p>\n<p>In fact, those pods are related to jobs created and executed in a background during the deployment of the SQL Server AG:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get jobs -n agdev\nNAME                      DESIRED   SUCCESSFUL   AGE\nmssql-initialize-mssql1   1         1            22h\nmssql-initialize-mssql2   1         1            22h\nmssql-initialize-mssql3   1         1            22h<\/pre>\n<p>&nbsp;<\/p>\n<p>Let\u2019s take a look at the mssql-initialize-mssql1 job:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl describe job -n agdev mssql-initialize-mssql1\nName:           mssql-initialize-mssql1\nNamespace:      agdev\nSelector:       controller-uid=cd481f3c-c5b5-11e8-a34a-0a09b8f01b34\nLabels:         controller-uid=cd481f3c-c5b5-11e8-a34a-0a09b8f01b34\n                job-name=mssql-initialize-mssql1\nAnnotations:    &lt;none&gt;\nParallelism:    1\nCompletions:    1\nStart Time:     Mon, 01 Oct 2018 22:08:45 +0200\nPods Statuses:  0 Running \/ 1 Succeeded \/ 0 Failed\nPod Template:\n  Labels:           controller-uid=cd481f3c-c5b5-11e8-a34a-0a09b8f01b34\n                    job-name=mssql-initialize-mssql1\n  Service Account:  mssql-initialize-mssql1\n  Containers:\n   mssql-initialize:\n    Image:      mcr.microsoft.com\/mssql\/ha:vNext-CTP2.0-ubuntu\n    Port:       &lt;none&gt;\n    Host Port:  &lt;none&gt;\n    Command:\n      \/mssql-server-k8s-init-sql\n    Environment:\n      MSSQL_K8S_NAMESPACE:              (v1:metadata.namespace)\n      MSSQL_K8S_SA_PASSWORD:           &lt;set to the key 'sapassword' in secret 'sql-secrets'&gt;  Optional: false\n      MSSQL_K8S_NUM_SQL_SERVERS:       1\n      MSSQL_K8S_SQL_POD_OWNER_UID:     cd13319a-c5b5-11e8-a34a-0a09b8f01b34\n      MSSQL_K8S_SQL_SERVER_NAME:       mssql1\n      MSSQL_K8S_SQL_POST_INIT_SCRIPT:\n      MSSQL_K8S_MASTER_KEY_PASSWORD:   &lt;set to the key 'masterkeypassword' in secret 'sql-secrets'&gt;  Optional: false\n    Mounts:                            &lt;none&gt;\n  Volumes:                             &lt;none&gt;\nEvents:                                &lt;none&gt;<\/pre>\n<p>&nbsp;<\/p>\n<p>These jobs are one-time initialization code that is executed when SQL Server and the AG is bootstrapped (thank you to <a href=\"https:\/\/twitter.com\/MihaelaBlendea\" target=\"_blank\" rel=\"noopener noreferrer\">@MihaelaBlendea<\/a> to give more details on this topic) through the mssql-server-k8s-init-sql command. This is likely something you may remove according to your context (if you daily deal with a lot of K8s jobs for example).<\/p>\n<p>Then, the deployment led to create 3 StatefulSets with their respective pods mssql1-0, mssql2-0 and mssql3-0. Each pod contains 2 containers as shown below for the mssql1-0 pod:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl describe pod -n agdev mssql1-0\nName:           mssql1-0\nNamespace:      agdev\nNode:           aks-nodepool1-78763348-1\/10.240.0.5\n\u2026\nStatus:         Running\nIP:             10.244.0.38\nControlled By:  StatefulSet\/mssql1\nContainers:\n  mssql-server:\n    Container ID:   docker:\/\/8e23cec873ea3d1ebd98f8f4f0ab0b11b840c54c17557d23817b9c21a863bb42\n    Image:          mcr.microsoft.com\/mssql\/server:vNext-CTP2.0-ubuntu\n    Image ID:       docker-pullable:\/\/mcr.microsoft.com\/mssql\/server@sha256:87e691e2e5f738fd64a427ebe935e4e5ccd631be1b4f66be1953c7450418c8c8\n    Ports:          1433\/TCP, 5022\/TCP\n    Host Ports:     0\/TCP, 0\/TCP\n    State:          Running\n      Started:      Mon, 01 Oct 2018 22:11:44 +0200\n    Ready:          True\n    Restart Count:  0\n    Liveness:       http-get http:\/\/:8080\/healthz delay=60s timeout=1s period=2s #success=1 #failure=3\n    Environment:\n      ACCEPT_EULA:        y\n      MSSQL_PID:          Developer\n      MSSQL_SA_PASSWORD:  &lt;set to the key 'initsapassword' in secret 'mssql1-statefulset-secret'&gt;  Optional: false\n      MSSQL_ENABLE_HADR:  1\n    Mounts:\n      \/var\/opt\/mssql from instance-root (rw)\n      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from no-api-access (ro)\n  mssql-ha-supervisor:\n    Container ID:  docker:\/\/f5a0d4d51a459752a2c509eb3ec7874d94586a7499201f559c9ad8281751e514\n    Image:         mcr.microsoft.com\/mssql\/ha:vNext-CTP2.0-ubuntu\n    Image ID:      docker-pullable:\/\/mcr.microsoft.com\/mssql\/ha@sha256:c5d20c8b34ea096a845de0222441304a14ad31a447d79904bafaf29f898704d0\n    Port:          8080\/TCP\n    Host Port:     0\/TCP\n    Command:\n      \/mssql-server-k8s-ag-agent-supervisor\n    State:          Running\n      Started:      Mon, 01 Oct 2018 22:11:45 +0200\n    Ready:          True\n    Restart Count:  0\n    Environment:\n      MSSQL_K8S_NAMESPACE:                         agdev (v1:metadata.namespace)\n      MSSQL_K8S_POD_NAME:                          mssql1-0 (v1:metadata.name)\n      MSSQL_K8S_SQL_SERVER_NAME:                   mssql1\n      MSSQL_K8S_POD_IP:                             (v1:status.podIP)\n      MSSQL_K8S_NODE_NAME:                          (v1:spec.nodeName)\n      MSSQL_K8S_MONITOR_POLICY:                    3\n      MSSQL_K8S_HEALTH_CONNECTION_REBOOT_TIMEOUT:\n      MSSQL_K8S_SKIP_AG_ANTI_AFFINITY:\n      MSSQL_K8S_MONITOR_PERIOD_SECONDS:\n      MSSQL_K8S_LEASE_DURATION_SECONDS:\n      MSSQL_K8S_RENEW_DEADLINE_SECONDS:\n      MSSQL_K8S_RETRY_PERIOD_SECONDS:\n      MSSQL_K8S_ACQUIRE_PERIOD_SECONDS:\n      MSSQL_K8S_SQL_WRITE_LEASE_PERIOD_SECONDS:\n    Mounts:\n      \/var\/run\/secrets\/kubernetes.io\/serviceaccount from mssql1-token-5zlkq (ro)\n\u2026.\nVolumes:\n  no-api-access:\n    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)\n    Medium:\n  instance-root:\n    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n    ClaimName:  mssql-data-1\n    ReadOnly:   false\n  mssql1-token-5zlkq:\n    Type:        Secret (a volume populated by a Secret)\n    SecretName:  mssql1-token-5zlkq\n    Optional:    false\n\u2026<\/pre>\n<p>&nbsp;<\/p>\n<p>We recognize the mssql-server and mssql-ha-supervisor container as stated to the <a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/linux\/sql-server-ag-kubernetes?view=sqlallproducts-allversions\" target=\"_blank\" rel=\"noopener noreferrer\">Microsoft documentation<\/a>. The mssql-server container is listening on the port 1433 (SQL engine) and 5022 (hadr point). Note the container includes a HTTP liveness probes (<strong>http-get http:\/\/:8080\/healthz delay=60s timeout=1s period=2s #success=1 #failure=3<\/strong>) to determine its health. Morever, the mssql-ha-supervisor container is self-explaining and aims to monitor the SQL Server instance if we refer to the environment variable names. I believe another blog post will be necessary to talk about it. Each SQL Server pod (meaning a SQL Server instance here that listen on the port 1433) is exposed to the external traffic by a dedicated service as shown below. External IPs are assigned to the K8s cluster load balancer services through the Azure Load Balancer (basic SKU).<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get svc -n agdev\nNAME                   TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)             AGE\nag1                    ClusterIP      None           &lt;none&gt;           1433\/TCP,5022\/TCP   23h\nmssql1                 LoadBalancer   10.0.43.216    xx.xx.xx.xxx    1433:31674\/TCP      23h\nmssql2                 LoadBalancer   10.0.28.27     xx.xx.xx.xxx    1433:32681\/TCP      23h\nmssql3                 LoadBalancer   10.0.137.244   xx.xx.xxx.xxx    1433:31152\/TCP      23h<\/pre>\n<p>&nbsp;<\/p>\n<ul>\n<li>The AG Services<\/li>\n<\/ul>\n<p>Finally, I only deployed the service corresponding to ag1-primary that connects to the primary replica. It is up to you to deploy other ones according to your context. In fact, the ag1-primary service acts as the AG listener in this new infrastructure.<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get svc -n agdev\nNAME          TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)             AGE\nag1           ClusterIP      None           &lt;none&gt;          1433\/TCP,5022\/TCP   23h\nag1-primary   LoadBalancer   10.0.32.104    xxx.xx.xx.xxx       1433:31960\/TCP      1m\nmssql1        LoadBalancer   10.0.43.216    xx.xx.xx.xxx   1433:31674\/TCP      23h\nmssql2        LoadBalancer   10.0.28.27     xx.xx.xx.xxx   1433:32681\/TCP      23h\nmssql3        LoadBalancer   10.0.137.244   xx.xx.xxx.xxx   1433:31152\/TCP      23h<\/pre>\n<p>&nbsp;<\/p>\n<p>So, it\u2019s time to connect to my availability group from the external IP of the ag1-primary service. I already add a test database to the availability group and here a picture of the situation:<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">-- groups info\nSELECT \n\tg.name as ag_name,\n\trgs.primary_replica, \n\trgs.primary_recovery_health_desc as recovery_health, \n\trgs.synchronization_health_desc as sync_health\nFROM sys.dm_hadr_availability_group_states as rgs\nJOIN sys.availability_groups AS g\n\t\t\t\t ON rgs.group_id = g.group_id\n\n-- replicas info\nSELECT \n\tg.name as ag_name,\n\tr.replica_server_name,\n\tr.availability_mode_desc as [availability_mode],\n\tr.failover_mode_desc as [failover_mode],\n\trs.is_local,\n\trs.role_desc as role,\n\trs.operational_state_desc as op_state,\n\trs.connected_state_desc as connect_state,\n\trs.synchronization_health_desc as sync_state,\n\trs.last_connect_error_number,\n\trs.last_connect_error_description\nFROM sys.dm_hadr_availability_replica_states AS rs\nJOIN sys.availability_replicas AS r\n\tON rs.replica_id = r.replica_id\nJOIN sys.availability_groups AS g\n\tON g.group_id = r.group_id\nORDER BY r.replica_server_name, rs.is_local;\n\n-- DB level          \nSELECT \n\tg.name as ag_name,\n\tr.replica_server_name,\n\tDB_NAME(drs.database_id) as [database_name],\n\tdrs.is_local,\n\tdrs.is_primary_replica,\n\tsynchronization_state_desc as sync_state,\n\tsynchronization_health_desc as sync_health,\n\tdatabase_state_desc as db_state\nFROM sys.dm_hadr_database_replica_states AS drs\n\t\t JOIN sys.availability_replicas AS r\n\t\t  ON r.replica_id = drs.replica_id\n\t\t JOIN sys.availability_groups AS g\n\t\t  ON g.group_id = drs.group_id\nORDER BY g.name, drs.is_primary_replica DESC;\nGO<\/pre>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-28208\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\" alt=\"blog 143 - 1 - AG config\" width=\"900\" height=\"211\" \/><\/a><\/p>\n<p>This is a common picture we may get with traditional availability group. Another way to identify the primary replica is going through the <strong>kubectl<\/strong> command pod and to filter by label as follows:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get pods -n agdev -l=\"role.ag.mssql.microsoft.com\/ag1\"=\"primary\"\nNAME       READY     STATUS    RESTARTS   AGE\nmssql1-0   2\/2       Running   0          1d<\/pre>\n<p>&nbsp;<\/p>\n<p>To finish, let\u2019s simulate the crash of the pod mssql1-0 and let\u2019s see what happens:<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl delete pod -n agdev mssql1-0\npod \"mssql1-0\" deleted\nkubectl get pods -n agdev\nNAME                              READY     STATUS        RESTARTS   AGE\nmssql-initialize-mssql1-plh8l     0\/1       Completed     0          1d\nmssql-initialize-mssql2-l6z8m     0\/1       Completed     0          1d\nmssql-initialize-mssql3-wrbkl     0\/1       Completed     0          1d\nmssql-operator-67447c4bd8-s6tbv   1\/1       Running       0          1d\nmssql1-0                          0\/2       Terminating   0          1d\nmssql2-0                          2\/2       Running       0          1d\nmssql3-0                          2\/2       Running       0          1d\n\n...\n\n$ kubectl get pods -n agdev\nNAME                              READY     STATUS              RESTARTS   AGE\nmssql-initialize-mssql1-plh8l     0\/1       Completed           0          1d\nmssql-initialize-mssql2-l6z8m     0\/1       Completed           0          1d\nmssql-initialize-mssql3-wrbkl     0\/1       Completed           0          1d\nmssql-operator-67447c4bd8-s6tbv   1\/1       Running             0          1d\nmssql1-0                          0\/2       ContainerCreating   0          9s\nmssql2-0                          2\/2       Running             0          1d\nmssql3-0                          2\/2       Running             0          1d\n\n...\n\n$ kubectl get pods -n agdev\nNAME                              READY     STATUS      RESTARTS   AGE\nmssql-initialize-mssql1-plh8l     0\/1       Completed   0          1d\nmssql-initialize-mssql2-l6z8m     0\/1       Completed   0          1d\nmssql-initialize-mssql3-wrbkl     0\/1       Completed   0          1d\nmssql-operator-67447c4bd8-s6tbv   1\/1       Running     0          1d\nmssql1-0                          2\/2       Running     0          2m\nmssql2-0                          2\/2       Running     0          1d\nmssql3-0                          2\/2       Running     0          1d<\/pre>\n<p>&nbsp;<\/p>\n<p>As expected, the controller detects the event and recreates accordingly an another mssql1-0 pod but that\u2019s not all. Firstly, let\u2019s say because we are concerned by StatefulSet the pod keeps the same identity. Then the controller performs also other tasks including failover the availability group to another pod and change the primary with the mssql3-0 pod as shown below. The label of this pod is updated to identify the new primary.<\/p>\n<pre class=\"brush: shell; gutter: true; first-line: 1\">$ kubectl get pods -n agdev -l=\"role.ag.mssql.microsoft.com\/ag1\"=\"primary\"\nNAME       READY     STATUS    RESTARTS   AGE\nmssql3-0   2\/2       Running   0          1d<\/pre>\n<p>&nbsp;<\/p>\n<p>This blog post was just an overview of what could be a SQL Server availability group on K8s. Obviously, there are a plenty of other interesting items to cover and to deep dive &#8230; probably in a near future. Stay tuned!<\/p>\n<p><span style=\"float: none;background-color: #ffffff;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;text-align: left;text-decoration: none;text-indent: 0px;text-transform: none\">By David Barbarin<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>A couple of weeks ago, Microsoft announced the first public CTP version of next SQL Server version\u00a0(CTP2). It is not a surprise, the SQL Server vNext becomes SQL Server 2019 and there are a plenty of enhancements as well as new features to discover. But for now, let\u2019s start with likely one of my favorites: [&hellip;]<\/p>\n","protected":false},"author":26,"featured_media":11749,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[955,198,1320,99],"tags":[297,1338,84,1365,89,51,1457],"type_dbi":[],"class_list":["post-11747","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud","category-database-management","category-devops","category-sql-server","tag-availability-groups","tag-azure","tag-high-availability","tag-k8s","tag-kubernetes","tag-sql-server","tag-sql-server-2019"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>First steps into SQL Server 2019 availability groups on K8s<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"First steps into SQL Server 2019 availability groups on K8s\" \/>\n<meta property=\"og:description\" content=\"A couple of weeks ago, Microsoft announced the first public CTP version of next SQL Server version\u00a0(CTP2). It is not a surprise, the SQL Server vNext becomes SQL Server 2019 and there are a plenty of enhancements as well as new features to discover. But for now, let\u2019s start with likely one of my favorites: [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-10-04T04:27:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"900\" \/>\n\t<meta property=\"og:image:height\" content=\"211\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Microsoft Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Microsoft Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"16 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\"},\"author\":{\"name\":\"Microsoft Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/bfab48333280d616e1170e7369df90a4\"},\"headline\":\"First steps into SQL Server 2019 availability groups on K8s\",\"datePublished\":\"2018-10-04T04:27:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\"},\"wordCount\":1392,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\",\"keywords\":[\"Availability groups\",\"Azure\",\"High availability\",\"k8s\",\"kubernetes\",\"SQL Server\",\"SQL Server 2019\"],\"articleSection\":[\"Cloud\",\"Database management\",\"DevOps\",\"SQL Server\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\",\"name\":\"First steps into SQL Server 2019 availability groups on K8s\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\",\"datePublished\":\"2018-10-04T04:27:29+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/bfab48333280d616e1170e7369df90a4\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\",\"contentUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg\",\"width\":900,\"height\":211},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"First steps into SQL Server 2019 availability groups on K8s\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/bfab48333280d616e1170e7369df90a4\",\"name\":\"Microsoft Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/c44a1a792c059f24055763aa77d80a244467f6eef724a8bd13db8d4a350b7a4c?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c44a1a792c059f24055763aa77d80a244467f6eef724a8bd13db8d4a350b7a4c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c44a1a792c059f24055763aa77d80a244467f6eef724a8bd13db8d4a350b7a4c?s=96&d=mm&r=g\",\"caption\":\"Microsoft Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/microsoft-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"First steps into SQL Server 2019 availability groups on K8s","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/","og_locale":"en_US","og_type":"article","og_title":"First steps into SQL Server 2019 availability groups on K8s","og_description":"A couple of weeks ago, Microsoft announced the first public CTP version of next SQL Server version\u00a0(CTP2). It is not a surprise, the SQL Server vNext becomes SQL Server 2019 and there are a plenty of enhancements as well as new features to discover. But for now, let\u2019s start with likely one of my favorites: [&hellip;]","og_url":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/","og_site_name":"dbi Blog","article_published_time":"2018-10-04T04:27:29+00:00","og_image":[{"width":900,"height":211,"url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg","type":"image\/jpeg"}],"author":"Microsoft Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Microsoft Team","Est. reading time":"16 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/"},"author":{"name":"Microsoft Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/bfab48333280d616e1170e7369df90a4"},"headline":"First steps into SQL Server 2019 availability groups on K8s","datePublished":"2018-10-04T04:27:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/"},"wordCount":1392,"commentCount":0,"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg","keywords":["Availability groups","Azure","High availability","k8s","kubernetes","SQL Server","SQL Server 2019"],"articleSection":["Cloud","Database management","DevOps","SQL Server"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/","url":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/","name":"First steps into SQL Server 2019 availability groups on K8s","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg","datePublished":"2018-10-04T04:27:29+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/bfab48333280d616e1170e7369df90a4"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#primaryimage","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/blog-143-1-AG-config-e1538633608375.jpg","width":900,"height":211},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/first-steps-into-sql-server-2019-availability-groups-on-k8s\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"First steps into SQL Server 2019 availability groups on K8s"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/bfab48333280d616e1170e7369df90a4","name":"Microsoft Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c44a1a792c059f24055763aa77d80a244467f6eef724a8bd13db8d4a350b7a4c?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c44a1a792c059f24055763aa77d80a244467f6eef724a8bd13db8d4a350b7a4c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c44a1a792c059f24055763aa77d80a244467f6eef724a8bd13db8d4a350b7a4c?s=96&d=mm&r=g","caption":"Microsoft Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/microsoft-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11747","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/26"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=11747"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11747\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media\/11749"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=11747"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=11747"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=11747"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=11747"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}