{"id":19723,"date":"2022-10-14T18:40:02","date_gmt":"2022-10-14T16:40:02","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=19723"},"modified":"2022-10-14T18:40:03","modified_gmt":"2022-10-14T16:40:03","slug":"kubernetes-metallb-in-depth","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/","title":{"rendered":"Kubernetes: MetalLB in-depth"},"content":{"rendered":"\n

By working on a project for one of our customer, I have investigated how MetalLB<\/a> works in details. This Kubernetes cluster is on-premises only and uses Nginx Ingress Controller<\/a> for processing incoming L7 and L4 traffic. If you want to deep dive into this technology then buckle up!<\/p>\n\n\n\n

Nginx Ingress Controller Service<\/h2>\n\n\n\n

When we want to give access to a Kubernetes cluster with several services from outside, the best practice is to use an ingress and create a service of type Load Balancer for processing those ingress rules.<\/p>\n\n\n\n

Nginx Ingress Controller is one of the options for doing this task and when it is setup you’ll see something like this:<\/p>\n\n\n

\n$ kubectl get svc -A|grep -i loadbalancer\nnginx-controller     ingress-nginx-ingress-nginx    LoadBalancer   172.10.10.10    10.0.0.220   443:31529\/TCP            4d3h\n<\/pre><\/div>\n\n\n
This is our Nginx service LoadBalancer that uses the external IP address of 10.0.0.220. This IP address is provided by MetalLB.<\/p>\n\n\n\n
MetalLB functionning<\/h2>\n\n\n\n
MetalLB has two components containerized into pods. One Controller and several Speakers. The Controller is the one responsible for giving the external IP address to the service LoadBalancer we have just seen above. There is one Speaker pod per node in the cluster that influences how the traffic is routed in the cluster.<\/p>\n\n\n\n
MetalLB routing can use two modes: L2 and BGP. L2 is the one we are using and in this mode the elected Speaker is responsible of the external IP address (which is then a floating IP shared between the Speakers) and will respond to ARP requests to it. The Speaker pod is a host-networked pod so it uses the IP address of the node it is hosted on. This node will then route the traffic. Knowing that, I wanted to look under the hood and check if we could find information about the elected Speaker and its associated node used to route the traffic. I also wanted to know if there was a load balancing at play and how it was working.<\/p>\n\n\n\n
Searching for L2 information<\/h2>\n\n\n\n
Finding the MAC address of the external IP address<\/h3>\n\n\n\n
The pod logs of the Speakers doesn’t reveal much except that there are some activities. However no details are logged.<\/p>\n\n\n\n
The first step is to connect to one node of the cluster and do the following:<\/p>\n\n\n
\n$ arp -a|grep 10.0.0.220\n$ ping 10.0.0.220\nPING 10.0.0.220 (10.0.0.220) 56(84) bytes of data.\n^C\n--- 10.0.0.220 ping statistics ---\n11 packets transmitted, 0 received, 100% packet loss, time 10267ms\n\narp -a|grep 10.0.0.220\nmetallb-ep.domain (10.0.0.220) at 00:50:56:08:79:fb [ether] on ens192\n<\/pre><\/div>\n\n\n
This node had no ARP entry for our external IP address so by pinging it we could force it to create that entry. MetalLB documentation in its troubleshooting<\/a> section doesn’t recommend ping but advice to use curl instead. In my case both worked. By doing a tcpdump in another terminal on this node at the same time we could see the ARP request and reply:<\/p>\n\n\n
\n# tcpdump -i ens192 -vv arp\ndropped privs to tcpdump\ntcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes\n08:39:13.799663 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has metallb-ep.domain tell node1, length 28\n08:39:13.800144 ARP, Ethernet (len 6), IPv4 (len 4), Reply metallb-ep.domain is-at 00:00:00:00:79:e9 (oui Unknown), length 46\n<\/pre><\/div>\n\n\n
From both the ARP table and the tcpdump trace we can see that the MAC address 00:00:00:00:79:e9 is associated with our external IP address. We know that this MAC address belongs to one of the node hosting the elected Speaker so let’s find it.<\/p>\n\n\n\n
Finding which node interface is used in the ARP reply<\/h3>\n\n\n\n
At this stage you may want to automate that interface search in all the nodes or use the information collected in a monitoring tool. In my case I had only 3 nodes with a metallb-speaker pod so I did the search manually:<\/p>\n\n\n
\n$ kubectl get po -A -owide |grep -i metal|grep -i speaker\nmetallb-controller   metallb-speaker-4w8wr                                     1\/1     Running   0             4d5h   10.0.0.1   node1   <none>           <none>\nmetallb-controller   metallb-speaker-6x9c4                                     1\/1     Running   0             4d5h   10.0.0.2   node2   <none>           <none>\nmetallb-controller   metallb-speaker-n4h8n                                     1\/1     Running   0             18m    10.0.0.3   node3   <none>           <none>\n<\/pre><\/div>\n\n\n
Then ssh to each of them:<\/p>\n\n\n
\n[node1 ~]$ ifconfig ens192\nens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500\n        inet 10.0.0.1  netmask 255.255.255.0  broadcast 10.0.0.255\n        ether 00:00:00:00:79:e9  txqueuelen 1000  (Ethernet)\n<\/pre><\/div>\n\n\n
Sometimes life is easy: The first node is the chosen one! I now know that all my traffic is routed through that node and that the elected Speaker is the pod on that node too.<\/p>\n\n\n\n
Load Balancing anyone?<\/h2>\n\n\n\n
Ok we could find which Speaker\/Node routes our traffic but can we see if another Speaker\/Node is chosen? And when? And how is it working? I’m glad you asked, I did the dirty work for you! <\/p>\n\n\n\n
I’m first going to use arping from a node on which there is no metallb-speaker pod at all. This is one step further from what is advised in the MetalLB troubleshooting section.<\/p>\n\n\n
\n[node4 ~]$ arping -I ens192 10.0.0.220\nARPING 10.0.0.220 from 10.0.0.4 ens192\nUnicast reply from 10.0.0.220 [00:00:00:00:79:E9]  1.033ms\nUnicast reply from 10.0.0.220 [00:00:00:00:7A:06]  1.051ms\nUnicast reply from 10.0.0.220 [00:00:00:00:79:FB]  1.228ms\nUnicast reply from 10.0.0.220 [00:00:00:00:79:FB]  1.059ms\nUnicast reply from 10.0.0.220 [00:00:00:00:79:FB]  1.089ms\nUnicast reply from 10.0.0.220 [00:00:00:00:79:FB]  1.177ms\nUnicast reply from 10.0.0.220 [00:00:00:00:79:FB]  1.067ms\n^C\n\n[node4 ~]$ arping -I ens192 10.0.0.220\nARPING 10.0.0.220 from 10.0.0.4 ens192\nUnicast reply from 10.0.0.220 [00:00:00:00:79:FB]  1.046ms\nUnicast reply from 10.0.0.220 [00:00:00:00:79:E9]  1.064ms\nUnicast reply from 10.0.0.220 [00:00:00:00:7A:06]  1.145ms\nUnicast reply from 10.0.0.220 [00:00:00:00:7A:06]  1.170ms\nUnicast reply from 10.0.0.220 [00:00:00:00:7A:06]  1.140ms\nUnicast reply from 10.0.0.220 [00:00:00:00:7A:06]  1.037ms\nUnicast reply from 10.0.0.220 [00:00:00:00:7A:06]  1.113ms\n^C\n<\/pre><\/div>\n\n\n
The behavior observed above is consistent during all my tests, the first 3 replies are the MAC address of the nodes hosting the 3 metallb-speaker and the last one continue responding. I understand it to be elected Speaker. When you run the arping again then the 3 first replies are in the different order and that looks like a load balancer mechanism.<\/p>\n\n\n\n
However that load balancing is working with arping which is designed to trigger an ARP request. When a node will communicate with the IP address of 10.0.0.220 it will use first its ARP table that contains the mapping between a MAC address and an IP address. If there is no entry then an ARP request will be triggered. In my case I had an entry already:<\/p>\n\n\n
\n[node4 ~]# date; arp -a|grep 10.0.0.220\nTue Oct 11 14:27:01 CEST 2022\nmetallb-ep.domain (10.0.0.220) at 00:00:00:00:7a:06 [ether] on ens192\n[node4 ~]# date; arp -a|grep 10.0.0.220\nTue Oct 11 14:27:50 CEST 2022\nmetallb-ep.domain (10.0.0.220) at 00:00:00:00:79:e9 [ether] on ens192\n[node4 ~]# date; arp -a|grep 10.0.0.220\nTue Oct 11 14:29:21 CEST 2022\nmetallb-ep.domain (10.0.0.220) at 00:00:00:00:7a:06 [ether] on ens192\n<\/pre><\/div>\n\n\n
On node4 we observe that the MAC address associated with our external IP address changes regularly to point to a different Speaker\/Node. That also looks like load balancing to me.<\/p>\n\n\n\n
Let’s sniff the ARP traffic on that node to see what’s going on:<\/p>\n\n\n
\n14:38:42.759258 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has metallb-ep.domain (Broadcast) tell metallb-ep.domain, length 46\n14:38:42.759317 ARP, Ethernet (len 6), IPv4 (len 4), Reply metallb-ep.domain is-at 00:00:00:00:79:fb (oui Unknown), length 46\n14:38:42.759323 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has mmetallb-ep.domain (Broadcast) tell metallb-ep.domain, length 46\n14:38:42.759392 ARP, Ethernet (len 6), IPv4 (len 4), Reply metallb-ep.domain is-at 00:00:00:00:7a:06 (oui Unknown), length 46\n14:38:42.760373 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has metallb-ep.domain (Broadcast) tell mmetallb-ep.domain, length 46\n14:38:42.760393 ARP, Ethernet (len 6), IPv4 (len 4), Reply metallb-ep.domain is-at 00:00:00:00:79:e9 (oui Unknown), length 46\n<\/pre><\/div>\n\n\n
There are 3 ARP requests broadcast (sent by each Speaker I guess) and their corresponding replies that are sent every 15 seconds or so.  When you observe the ARP table at the same time you see a change so those replies are used like some kind of gratuitous ARP to change the MAC address associated with the external IP address. Thus a different node will be used for that IP address so that still looks very much like load balancing to me.<\/p>\n\n\n\n
Last but not least<\/h2>\n\n\n\n
As I’ve started to push, and I could experiment – DevOps style – with a cluster not in production, I finally wanted to know what are the effects of killing a metallb-speaker pod.<\/p>\n\n\n\n
What I’ve observed is that if I kill the Speaker pod hosted on the node that I have the MAC address in my ARP table (so the elected Speaker), then a bunch of ARP request broadcast as shown above is sent immediately. However my node doesn’t necessarily changes its ARP table entry. The pod comes up in less than 15s and so the same one could be elected again apparently. I’ve read that a fail over has around 10s of recovery time so that fits with what I’ve observed.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
With L2 node, only one node is used at a time to route the traffic. I read that load balancing was not possible in this mode. However, even if it is not the best way to achieve it, I’ve observed some load balancing mechanism not only when the elected Speaker is deleted but also at a regular interval of tens of seconds.<\/p>\n\n\n\n
I haven’t found any explanation of those internals anywhere on the net so this is my contribution to build upon and go further on that topic.<\/p>\n\n\n\n
Note that we have other older clusters where the MAC address of the Speaker\/Node doesn’t change so the load balancing observation described in this blog doesn’t apply. The MetalLB version in those clusters is older than the one I’ve tested so this load balancing behaviour may have been improved, I’ll have to do further investigation.<\/p>\n\n\n\n
Note also that the IP addresses, Ethernet Adresses and Host names have been anonymized.<\/p>\n","protected":false},"excerpt":{"rendered":"
By working on a project for one of our customer, I have investigated how MetalLB works in details. This Kubernetes cluster is on-premises only and uses Nginx Ingress Controller for processing incoming L7 and L4 traffic. If you want to deep dive into this technology then buckle up! Nginx Ingress Controller Service When we want […]<\/p>\n","protected":false},"author":109,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1320,1522],"tags":[2723,1671,2634,2722,2704],"type_dbi":[],"class_list":["post-19723","post","type-post","status-publish","format-standard","hentry","category-devops","category-kubernetes","tag-arp","tag-ingress","tag-kubernetes-2","tag-metallb","tag-nginx"],"acf":[],"yoast_head":"\nKubernetes: MetalLB in-depth - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kubernetes: MetalLB in-depth\" \/>\n<meta property=\"og:description\" content=\"By working on a project for one of our customer, I have investigated how MetalLB works in details. This Kubernetes cluster is on-premises only and uses Nginx Ingress Controller for processing incoming L7 and L4 traffic. If you want to deep dive into this technology then buckle up! Nginx Ingress Controller Service When we want […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-14T16:40:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-10-14T16:40:03+00:00\" \/>\n<meta name=\"author\" content=\"DevOps\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DevOps\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/\"},\"author\":{\"name\":\"DevOps\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/4cd1b5f8a3de93f05a16ab8d7d2b7735\"},\"headline\":\"Kubernetes: MetalLB in-depth\",\"datePublished\":\"2022-10-14T16:40:02+00:00\",\"dateModified\":\"2022-10-14T16:40:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/\"},\"wordCount\":1185,\"commentCount\":0,\"keywords\":[\"arp\",\"Ingress\",\"kubernetes\",\"metallb\",\"Nginx\"],\"articleSection\":[\"DevOps\",\"Kubernetes\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/\",\"name\":\"Kubernetes: MetalLB in-depth - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"datePublished\":\"2022-10-14T16:40:02+00:00\",\"dateModified\":\"2022-10-14T16:40:03+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/4cd1b5f8a3de93f05a16ab8d7d2b7735\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kubernetes-metallb-in-depth\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kubernetes: MetalLB in-depth\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/4cd1b5f8a3de93f05a16ab8d7d2b7735\",\"name\":\"DevOps\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"caption\":\"DevOps\"},\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/devops\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Kubernetes: MetalLB in-depth - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/","og_locale":"en_US","og_type":"article","og_title":"Kubernetes: MetalLB in-depth","og_description":"By working on a project for one of our customer, I have investigated how MetalLB works in details. This Kubernetes cluster is on-premises only and uses Nginx Ingress Controller for processing incoming L7 and L4 traffic. If you want to deep dive into this technology then buckle up! Nginx Ingress Controller Service When we want […]","og_url":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/","og_site_name":"dbi Blog","article_published_time":"2022-10-14T16:40:02+00:00","article_modified_time":"2022-10-14T16:40:03+00:00","author":"DevOps","twitter_card":"summary_large_image","twitter_misc":{"Written by":"DevOps","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/"},"author":{"name":"DevOps","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735"},"headline":"Kubernetes: MetalLB in-depth","datePublished":"2022-10-14T16:40:02+00:00","dateModified":"2022-10-14T16:40:03+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/"},"wordCount":1185,"commentCount":0,"keywords":["arp","Ingress","kubernetes","metallb","Nginx"],"articleSection":["DevOps","Kubernetes"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/","url":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/","name":"Kubernetes: MetalLB in-depth - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2022-10-14T16:40:02+00:00","dateModified":"2022-10-14T16:40:03+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/kubernetes-metallb-in-depth\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Kubernetes: MetalLB in-depth"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735","name":"DevOps","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","caption":"DevOps"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/devops\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/19723","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/109"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=19723"}],"version-history":[{"count":40,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/19723\/revisions"}],"predecessor-version":[{"id":19804,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/19723\/revisions\/19804"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=19723"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=19723"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=19723"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=19723"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}