{"id":17103,"date":"2022-02-04T20:25:46","date_gmt":"2022-02-04T19:25:46","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/"},"modified":"2022-05-31T16:33:44","modified_gmt":"2022-05-31T14:33:44","slug":"creating-kvm-database-system-on-separate-vlan-network-on-oda","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/","title":{"rendered":"Creating KVM Database System on separate VLAN network on ODA"},"content":{"rendered":"

Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how to create additional network on the ODA and how to create DB System on separate VLAN interface. The principle would be of course the same if we would like to create KVM Virtual Machines (compute instance) on separate VLAN.<\/p>\n

<\/p>\n

Configuration description<\/h3>\n

For my test I have got an ODA X8-2M with a quad-port 10GBase-T network interface. I’m running ODA version 19.13.
\nOn my network card, the 2 first ports p7p1 and p7p2 will be assigned to btbond1 and the 2 next ports p7p3 and p7p4 will be assigned to my second bonding interface btbond2. ODA are configured by default with active-backup mode without LACP for all bonding interfaces. This is automatically designed by the ODA and can not be changed. More over we need to keep in mind that on an appliance we will never manually change the Linux network scripts. All network configuration changes needs to be done with odacli.<\/p>\n

btbond1 is used for my main network and we will use btbond2 to add additionnal networks.<\/p>\n

The 2 additional networks are :
\n10.38.0.1\/24 VLAN id 38
\n10.39.0.1\/24 VLAN id 39<\/p>\n

Checking network interface<\/h3>\n

With ethtool I can check and see that the 2 p7p3 and p7p4 ports are twisted pair and connected to the network :<\/p>\n

\n[root@dbi-oda-x8 ~]# ethtool p7p3 | grep -iE \"(bportb|detected)\"\n\tPort: Twisted Pair\n\tLink detected: yes\n\n[root@dbi-oda-x8 ~]# ethtool p7p4 | grep -iE \"(bportb|detected)\"\n\tPort: Twisted Pair\n\tLink detected: yes\n<\/pre>\n
<\/br><\/p>\n
I can see that both those interfaces are assigned to btbond2 interface :<\/p>\n
\n[root@dbi-oda-x8 ~]# grep MASTER \/etc\/sysconfig\/network-scripts\/ifcfg-p7p3\nMASTER=btbond2\n\n[root@dbi-oda-x8 ~]# grep MASTER \/etc\/sysconfig\/network-scripts\/ifcfg-p7p4\nMASTER=btbond2\n<\/pre>\n
<\/br><\/p>\n
Checking existing and default ODA configuration<\/h3>\n
After reimaging an ODA we would have the below default configuration.<\/p>\n
\nroot@dbi-oda-x8 ~]# odacli list-networks\n\nID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks\n-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------\nbb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]\n95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]\ne77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]\n\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\n\n[root@dbi-oda-x8 ~]#\n<\/pre>\n
<\/br><\/p>\n
To use a network interface with the VMs, either a DB System or a Compute instance, the network interface needs to be created in the vnetworks.<\/p>\n
VLAN Tagged versus Untagged<\/h3>\n
If a port on a switch is configured as tagged (trunk port in cisco world), the connected equipment (here our ODA) is VLAN aware, and would need to be tagged as well. The port is enabled for VLAN tagging. Purpose is to pass traffic for multiple VLAN’s. The ethernet frame is enclosing an additional VLAN header. The connected equipment will add the VLAN information in the ethernet frame.<\/p>\n
If a port is configured as untagged (access port in cisco world), the connected equipment (here our ODA) does not know anything about in which VLAN it is and does not care about it. The switch will manage it on its own. The port does not tag and only accepts a single VLAN. The ethernet frame coming to the connected equipment will not have any VLAN header. <\/p>\n
This is explained in the 802.1Q standard.<\/p>\n
Most of the time trunk ports will link switches and access ports will link to end devices, albeit we would like to use several VLAN network on the same ODA network interface.<\/p>\n
Creating an additional untagged network<\/h3>\n
Let’s create an additional untagged physical network on the Bare Metal itself. We will create it on btbond2 interface. Untagged means that the ODA will not add VLAN information in the Ethernet frame.
\nThe option -t bond would be the default one and would here not be required.<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m untagged1 -s 255.255.255.0 -g 10.38.0.1 -t bond\n{\n  \"jobId\" : \"9a4476dd-b955-433b-9463-377f66ab737a\",\n  \"status\" : \"Created\",\n  \"message\" : null,\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"February 04, 2022 13:48:12 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Rac Network service creation with name untagged1\",\n  \"updatedTime\" : \"February 04, 2022 13:48:12 PM CET\"\n}\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i 9a4476dd-b955-433b-9463-377f66ab737a\n\nJob details\n----------------------------------------------------------------\n                     ID:  9a4476dd-b955-433b-9463-377f66ab737a\n            Description:  Rac Network service creation with name untagged1\n                 Status:  Success\n                Created:  February 4, 2022 1:48:12 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nSetting network                          February 4, 2022 1:48:12 PM CET     February 4, 2022 1:48:18 PM CET     Success\nSetting up Network                       February 4, 2022 1:48:12 PM CET     February 4, 2022 1:48:12 PM CET     Success\nrestart network interface btbond2        February 4, 2022 1:48:12 PM CET     February 4, 2022 1:48:18 PM CET     Success\n\n[root@dbi-oda-x8 ~]# odacli list-networks\n\nID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks\n-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------\nbb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]\n95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]\ne77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]\n6171aad7-e247-4b08-a56e-83bdedd74af1   untagged1            btbond2      BOND            255.255.255.0      10.38.0.1                   [IP Address on node0: 10.38.0.10]\n\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\n<\/pre>\n
<\/br><\/p>\n
I can see that our new network exists in the network list but not in the virtual network of course.<\/p>\n
I can also check and see that our btbond2 interface has been configured by odacli as an untagged bonding interface with the appropriate information.<\/p>\n
\n[root@dbi-oda-x8 ~]# ip addr sh btbond2\n9: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n    link\/ether 3c:fd:fe:92:80:1a brd ff:ff:ff:ff:ff:ff\n    inet 10.38.0.10\/24 brd 10.38.0.255 scope global btbond2\n       valid_lft forever preferred_lft forever\n\n[root@dbi-oda-x8 ~]# ls -ltrh \/etc\/sysconfig\/network-scripts\/ifcfg*btbond2*\n-rw-r--r--. 1 root root 262 Feb  4 13:48 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2\n\n[root@dbi-oda-x8 ~]# cat \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2\n#This file was created by ODA. Do not edit.\nNETMASK=255.255.255.0\nGATEWAY=10.38.0.1\nBOOTPROTO=none\nPEERDNS=no\nDEVICE=btbond2\nONBOOT=yes\nNM_CONTROLLED=no\nIPADDR=10.38.0.10\nBONDING_OPTS=\"mode=active-backup miimon=100 primary=p7p3\"\nIPV6INIT=no\nUSERCTL=no\nTYPE=BOND\n<\/pre>\n
<\/br><\/p>\n
Of course it is not possible to create another untagged network on btbond2 interface knowing there is already one existing :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bond\nDCS-10001:Internal error encountered: nicnamebtbond2 already exists in the networks list .. .\n<\/pre>\n
<\/br><\/p>\n
Creating an additional tagged network would also not be possible. Note the option -t and -v to create tagged networks on the ODA :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged2 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39\nDCS-10001:Internal error encountered: Creating vlan in the interface btbond2 is not allowed. Physical network untagged1 already exists in interface btbond2.\n<\/pre>\n
<\/br><\/p>\n
Let’s delete our additional untagged network<\/h3>\n
\n[root@dbi-oda-x8 ~]# odacli delete-network -m untagged1\n{\n  \"jobId\" : \"3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\",\n  \"status\" : \"Running\",\n  \"message\" : null,\n  \"reports\" : [ {\n    \"taskId\" : \"TaskSequential_10041\",\n    \"taskName\" : \"deleting network\",\n    \"taskResult\" : \"\",\n    \"startTime\" : \"February 04, 2022 14:08:32 PM CET\",\n    \"endTime\" : \"February 04, 2022 14:08:32 PM CET\",\n    \"status\" : \"Running\",\n    \"taskDescription\" : null,\n    \"parentTaskId\" : \"TaskSequential_10039\",\n    \"jobId\" : \"3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\",\n    \"tags\" : [ ],\n    \"reportLevel\" : \"Info\",\n    \"updatedTime\" : \"February 04, 2022 14:08:32 PM CET\"\n  }, {\n    \"taskId\" : \"TaskZJsonRpcExt_10045\",\n    \"taskName\" : \"Setting up Network\",\n    \"taskResult\" : \"Network setup success\",\n    \"startTime\" : \"February 04, 2022 14:08:32 PM CET\",\n    \"endTime\" : \"February 04, 2022 14:08:32 PM CET\",\n    \"status\" : \"Success\",\n    \"taskDescription\" : null,\n    \"parentTaskId\" : \"TaskSequential_10041\",\n    \"jobId\" : \"3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\",\n    \"tags\" : [ ],\n    \"reportLevel\" : \"Info\",\n    \"updatedTime\" : \"February 04, 2022 14:08:32 PM CET\"\n  }, {\n    \"taskId\" : \"TaskZJsonRpcExt_10048\",\n    \"taskName\" : \"restart network interface btbond2\",\n    \"taskResult\" : \"\",\n    \"startTime\" : \"February 04, 2022 14:08:32 PM CET\",\n    \"endTime\" : \"February 04, 2022 14:08:32 PM CET\",\n    \"status\" : \"Running\",\n    \"taskDescription\" : null,\n    \"parentTaskId\" : \"TaskSequential_10047\",\n    \"jobId\" : \"3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\",\n    \"tags\" : [ ],\n    \"reportLevel\" : \"Info\",\n    \"updatedTime\" : \"February 04, 2022 14:08:32 PM CET\"\n  } ],\n  \"createTimestamp\" : \"February 04, 2022 14:08:32 PM CET\",\n  \"resourceList\" : [ {\n    \"resourceId\" : \"6171aad7-e247-4b08-a56e-83bdedd74af1\",\n    \"resourceType\" : null,\n    \"resourceNewType\" : \"Network\",\n    \"jobId\" : \"3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\",\n    \"updatedTime\" : null\n  } ],\n  \"description\" : \"Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1\",\n  \"updatedTime\" : \"February 04, 2022 14:08:32 PM CET\"\n}\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i 3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\n\nJob details\n----------------------------------------------------------------\n                     ID:  3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57\n            Description:  Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1\n                 Status:  Success\n                Created:  February 4, 2022 2:08:32 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\ndeleting network                         February 4, 2022 2:08:32 PM CET     February 4, 2022 2:08:34 PM CET     Success\nSetting up Network                       February 4, 2022 2:08:32 PM CET     February 4, 2022 2:08:32 PM CET     Success\nrestart network interface btbond2        February 4, 2022 2:08:32 PM CET     February 4, 2022 2:08:34 PM CET     Success\n<\/pre>\n
<\/br><\/p>\n
Network has been deleted :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-networks\n\nID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks\n-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------\nbb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]\n95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]\ne77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]\n<\/pre>\n
<\/br><\/p>\n
Creating additional tagged network<\/h3>\n
Let’s create the first tagged network on btbond2 interface :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m tagged38 -s 255.255.255.0 -g 10.38.0.1 -t VLAN -v 38\n{\n  \"jobId\" : \"b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc\",\n  \"status\" : \"Created\",\n  \"message\" : null,\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"February 04, 2022 14:40:28 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Rac Network service creation with name tagged38\",\n  \"updatedTime\" : \"February 04, 2022 14:40:28 PM CET\"\n}\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc\n\nJob details\n----------------------------------------------------------------\n                     ID:  b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc\n            Description:  Rac Network service creation with name tagged38\n                 Status:  Success\n                Created:  February 4, 2022 2:40:28 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nSetting network                          February 4, 2022 2:40:28 PM CET     February 4, 2022 2:40:33 PM CET     Success\nSetting up Vlan                          February 4, 2022 2:40:28 PM CET     February 4, 2022 2:40:33 PM CET     Success\n<\/pre>\n
<\/br><\/p>\n
The tagged network has been created :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-networks\n\nID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks\n-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------\nbb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]\n95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]\ne77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]\nfbe05bfa-636e-4f9c-a348-c59ba23e2296   tagged38             btbond2.38   VLAN            255.255.255.0      10.38.0.1          38       [IP Address on node0: 10.38.0.10]\n<\/pre>\n
<\/br><\/p>\n
And we can see that respective tagged network interface has now been created on the Linux operating system side (note the <btbond>.<vlan_id>) :<\/p>\n
\n[root@dbi-oda-x8 ~]# ls -ltrh \/etc\/sysconfig\/network-scripts\/ifcfg*btbond2*\n-rw-r--r--. 1 root root 239 Feb  4 14:38 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2\n-rw-r--r--  1 root root 520 Feb  4 14:40 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.38\n\n[root@dbi-oda-x8 ~]# cat \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.38\n#ODA_VLAN_CONFIG ===\n#ODA_VLAN_CONFIG Name=tagged38\n#ODA_VLAN_CONFIG VlanId=38\n#ODA_VLAN_CONFIG VlanInterface=btbond2\n#ODA_VLAN_CONFIG Type=VlanType\n#ODA_VLAN_CONFIG VlanSetupType=Other\n#ODA_VLAN_CONFIG VlanIpAddr=10.38.0.10\n#ODA_VLAN_CONFIG VlanNetmask=255.255.255.0\n#ODA_VLAN_CONFIG VlanGateway=10.38.0.1\n#ODA_VLAN_CONFIG NodeNum=0\n#=== DO NOT EDIT ANYTHING ABOVE THIS LINE ===\nDEVICE=btbond2.38\nBOOTPROTO=none\nONBOOT=yes\nVLAN=yes\nNM_CONTROLLED=no\nDEFROUTE=no\nIPADDR=10.38.0.10\nNETMASK=255.255.255.0\nGATEWAY=10.38.0.1\n[root@dbi-oda-x8 ~]#\n<\/pre>\n
<\/br><\/p>\n
And I can reach the ODA on the new network :<\/p>\n
\nC:Users>ping 10.38.0.10\n\nPinging 10.38.0.10 with 32 bytes of data:\nReply from 10.38.0.10: bytes=32 time=1ms TTL=63\nReply from 10.38.0.10: bytes=32 time<1ms TTL=63\nReply from 10.38.0.10: bytes=32 time<1ms TTL=63\nReply from 10.38.0.10: bytes=32 time<1ms TTL=63\n\nPing statistics for 10.38.0.10:\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\nApproximate round trip times in milli-seconds:\n    Minimum = 0ms, Maximum = 1ms, Average = 0ms\n<\/pre>\n
<\/br><\/p>\n
The ODA will not permit to create any untagged additional network on the btbond2 interface knowing one tagged network already exists :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bond\nDCS-10001:Internal error encountered: Creating non-VLAN typed network on the interface btbond2 is not allowed. VLAN tagged38 already exists in interface btbond2.\n<\/pre>\n
<\/br><\/p>\n
Let’s create the second tagged network on the same btbond2 interface :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged39 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39\n{\n  \"jobId\" : \"e0452663-544c-47c4-8b9b-5fbe6e0a5cd9\",\n  \"status\" : \"Created\",\n  \"message\" : null,\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"February 04, 2022 14:45:18 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Rac Network service creation with name tagged39\",\n  \"updatedTime\" : \"February 04, 2022 14:45:18 PM CET\"\n}\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i e0452663-544c-47c4-8b9b-5fbe6e0a5cd9\n\nJob details\n----------------------------------------------------------------\n                     ID:  e0452663-544c-47c4-8b9b-5fbe6e0a5cd9\n            Description:  Rac Network service creation with name tagged39\n                 Status:  Success\n                Created:  February 4, 2022 2:45:18 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nSetting network                          February 4, 2022 2:45:18 PM CET     February 4, 2022 2:45:23 PM CET     Success\nSetting up Vlan                          February 4, 2022 2:45:18 PM CET     February 4, 2022 2:45:23 PM CET     Success\n<\/pre>\n
<\/br><\/p>\n
New tagged network has been created and we now have 2 tagged network running on the same btbond2 interface :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-networks\n\nID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks\n-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------\nbb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]\n95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]\ne77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]\nfbe05bfa-636e-4f9c-a348-c59ba23e2296   tagged38             btbond2.38   VLAN            255.255.255.0      10.38.0.1          38       [IP Address on node0: 10.38.0.10]\ndc11ecbe-c4be-4e18-9c37-9f6360b37ee1   tagged39             btbond2.39   VLAN            255.255.255.0      10.39.0.1          39       [IP Address on node0: 10.39.0.10]\n<\/pre>\n
<\/br><\/p>\n
The respective new tagged interface has been configured by odacli on the linux operating system :<\/p>\n
\n[root@dbi-oda-x8 ~]# ls -ltrh \/etc\/sysconfig\/network-scripts\/ifcfg*btbond2*\n-rw-r--r--. 1 root root 239 Feb  4 14:38 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2\n-rw-r--r--  1 root root 520 Feb  4 14:40 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.38\n-rw-r--r--  1 root root 520 Feb  4 14:45 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.39\n\n[root@dbi-oda-x8 ~]# cat \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.39\n#ODA_VLAN_CONFIG ===\n#ODA_VLAN_CONFIG Name=tagged39\n#ODA_VLAN_CONFIG VlanId=39\n#ODA_VLAN_CONFIG VlanInterface=btbond2\n#ODA_VLAN_CONFIG Type=VlanType\n#ODA_VLAN_CONFIG VlanSetupType=Other\n#ODA_VLAN_CONFIG VlanIpAddr=10.39.0.10\n#ODA_VLAN_CONFIG VlanNetmask=255.255.255.0\n#ODA_VLAN_CONFIG VlanGateway=10.39.0.1\n#ODA_VLAN_CONFIG NodeNum=0\n#=== DO NOT EDIT ANYTHING ABOVE THIS LINE ===\nDEVICE=btbond2.39\nBOOTPROTO=none\nONBOOT=yes\nVLAN=yes\nNM_CONTROLLED=no\nDEFROUTE=no\nIPADDR=10.39.0.10\nNETMASK=255.255.255.0\nGATEWAY=10.39.0.1\n<\/pre>\n
<\/br><\/p>\n
And I can ping my new network as well :<\/p>\n
\nC:Users>ping 10.39.0.10\n\nPinging 10.39.0.10 with 32 bytes of data:\nReply from 10.39.0.10: bytes=32 time=2ms TTL=63\nReply from 10.39.0.10: bytes=32 time<1ms TTL=63\nReply from 10.39.0.10: bytes=32 time<1ms TTL=63\nReply from 10.39.0.10: bytes=32 time<1ms TTL=63\n\nPing statistics for 10.39.0.10:\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\nApproximate round trip times in milli-seconds:\n    Minimum = 0ms, Maximum = 2ms, Average = 0ms\n<\/pre>\n
<\/br><\/p>\n
I can now as well check my global btbond2 configuration on the linux side :<\/p>\n
\n[root@dbi-oda-x8 ~]# ip addr sh | grep -iE \"(btbond2|btbond2.38|btbond2.39)\"\n4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n9: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n57: btbond2.38@btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n    inet 10.38.0.10\/24 brd 10.38.0.255 scope global btbond2.38\n58: btbond2.39@btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n    inet 10.39.0.10\/24 brd 10.39.0.255 scope global btbond2.39\n[root@dbi-oda-x8 ~]#\n<\/pre>\n
<\/br><\/p>\n
Creating virtual networks<\/h3>\n
Knowing I need to use the networks for the next DB Systems I have deleted the created physical networks and I’m going to create the 2 same tagged networks as virtual networks on the btbond2 interface :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged38 -if btbond2 -t bridged-vlan -ip 10.38.0.10 -nm 255.255.255.0 -vlan 38 -gw 10.38.0.1\n\nJob details\n----------------------------------------------------------------\n                     ID:  eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1\n            Description:  vNetwork tagged38 creation\n                 Status:  Created\n                Created:  February 4, 2022 3:19:02 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1\n\nJob details\n----------------------------------------------------------------\n                     ID:  eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1\n            Description:  vNetwork tagged38 creation\n                 Status:  Success\n                Created:  February 4, 2022 3:19:02 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nValidate Virtual Network doesn‘t exist   February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success\nValidate interface to use exists         February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success\nValidate interfaces to create not exist  February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success\nCreate bridge                            February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success\nCreate VLAN                              February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success\nBring up VLAN                            February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:07 PM CET     Success\nCreate metadata                          February 4, 2022 3:19:07 PM CET     February 4, 2022 3:19:07 PM CET     Success\nPersist metadata                         February 4, 2022 3:19:07 PM CET     February 4, 2022 3:19:07 PM CET     Success\n\n[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -ip 10.39.0.10 -nm 255.255.255.0 -vlan 39 -gw 10.39.0.1\n\nJob details\n----------------------------------------------------------------\n                     ID:  15cf889f-9c6e-4e0c-a676-b2203e40cfd2\n            Description:  vNetwork tagged39 creation\n                 Status:  Created\n                Created:  February 4, 2022 3:19:40 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i 15cf889f-9c6e-4e0c-a676-b2203e40cfd2\n\nJob details\n----------------------------------------------------------------\n                     ID:  15cf889f-9c6e-4e0c-a676-b2203e40cfd2\n            Description:  vNetwork tagged39 creation\n                 Status:  Success\n                Created:  February 4, 2022 3:19:40 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nValidate Virtual Network doesn‘t exist   February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success\nValidate interface to use exists         February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success\nValidate interfaces to create not exist  February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success\nCreate bridge                            February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success\nCreate VLAN                              February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success\nBring up VLAN                            February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:45 PM CET     Success\nCreate metadata                          February 4, 2022 3:19:45 PM CET     February 4, 2022 3:19:45 PM CET     Success\nPersist metadata                         February 4, 2022 3:19:45 PM CET     February 4, 2022 3:19:45 PM CET     Success\n<\/pre>\n
<\/br><\/p>\n
My 2 new tagged networks exists as virtual network and not physical networks :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-networks\n\nID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks\n-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------\nbb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]\n95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]\ne77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]\n\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\ntagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-04 15:19:45 CET  2022-02-04 15:19:45 CET\ntagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\n<\/pre>\n
<\/br><\/p>\n
Tagged network interfaces have been created by the dcs-agent on the linux operating side :<\/p>\n
\n[root@dbi-oda-x8 ~]# ls -ltrh \/etc\/sysconfig\/network-scripts\/ifcfg*btbond2*\n-rw-r--r--. 1 root root 239 Feb  4 14:38 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2\n-rw-r--r--  1 root root 145 Feb  4 15:19 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.38\n-rw-r--r--  1 root root 145 Feb  4 15:19 \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.39\n\n[root@dbi-oda-x8 ~]# cat \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.38\n#This file was created by ODA. Do not edit.\nDEVICE=btbond2.38\nONBOOT=yes\nNM_CONTROLLED=no\nBOOTPROTO=none\nVLAN=yes\nONPARENT=yes\nBRIDGE=brtagged38\n\n[root@dbi-oda-x8 ~]# cat \/etc\/sysconfig\/network-scripts\/ifcfg-btbond2.39\n#This file was created by ODA. Do not edit.\nDEVICE=btbond2.39\nONBOOT=yes\nNM_CONTROLLED=no\nBOOTPROTO=none\nVLAN=yes\nONPARENT=yes\nBRIDGE=brtagged39\n<\/pre>\n
<\/br><\/p>\n
On the linux side, I can see that the respective IP addresses have not been assigned to the physical interfaces :<\/p>\n
\n[root@dbi-oda-x8 ~]# ip addr sh | grep -iE \"(btbond2|btbond2.38|btbond2.39)\"\n4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n9: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n60: btbond2.38@btbond2:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000\n62: btbond2.39@btbond2:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000\n<\/pre>\n
<\/br><\/p>\n
But on new virtual interfaces linked to the tagged btbond2 interfaces :<\/p>\n
\n[root@dbi-oda-x8 ~]# ip addr sh | grep -iE \"(tagged38|tagged39)\"\n59: brtagged38:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n    inet 10.38.0.10\/24 brd 10.38.0.255 scope global brtagged38\n60: btbond2.38@btbond2:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000\n61: brtagged39:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n    inet 10.39.0.10\/24 brd 10.39.0.255 scope global brtagged39\n62: btbond2.39@btbond2:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000\n<\/pre>\n
<\/br><\/p>\n
And I can ping them :<\/p>\n
\nC:Users>ping 10.38.0.10\n\nPinging 10.38.0.10 with 32 bytes of data:\nReply from 10.38.0.10: bytes=32 time=2ms TTL=63\nReply from 10.38.0.10: bytes=32 time<1ms TTL=63\nReply from 10.38.0.10: bytes=32 time<1ms TTL=63\n\nC:Users>ping 10.39.0.10\n\nPinging 10.39.0.10 with 32 bytes of data:\nReply from 10.39.0.10: bytes=32 time=2ms TTL=63\nReply from 10.39.0.10: bytes=32 time<1ms TTL=63\nReply from 10.39.0.10: bytes=32 time=1ms TTL=63\nReply from 10.39.0.10: bytes=32 time<1ms TTL=63\n\nPing statistics for 10.39.0.10:\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\nApproximate round trip times in milli-seconds:\n    Minimum = 0ms, Maximum = 2ms, Average = 0ms\n<\/pre>\n
<\/br><\/p>\n
I can see the network bridge information from the operating system as well :<\/p>\n
\n[root@dbi-oda-x8 ~]# brctl show\nbridge name\t  bridge id\t\t        STP enabled\t       interfaces\nbrtagged38\t\t8000.5254003bdc76\t  no\t\t             btbond2.38\n\t\t\t\t\t\t\t                                       vnet2\nbrtagged39\t\t8000.52540060eb57\t  no\t\t             btbond2.39\n\t\t\t\t\t\t\t                                       vnet4\nprivasm\t\t    8000.5a8241cdb508\t  no\t\t             priv0.100\n\t\t\t\t\t\t\t                                       vnet1\n\t\t\t\t\t\t\t                                       vnet3\npubnet\t\t    8000.3cfdfe928018\t  no\t               btbond1\n\t\t\t\t\t\t\t                                       vnet0\nvirbr0\t\t    8000.525400dc8c09\t  yes\t\t             virbr0-nic\n<\/pre>\n
<\/br><\/p>\n
Creating 2 DB Systems using respectively tagged38 and tagged39 virtual networks<\/h3>\n
My repository has already been updated with the KVM DB System Image :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli describe-dbsystem-image\nDB System Image details\n--------------------------------------------------------------------------------\nComponent Name        Supported Versions    Available Versions\n--------------------  --------------------  --------------------\n\nDBVM                  19.13.0.0.0           19.13.0.0.0\n\nGI                    19.13.0.0.211019      19.13.0.0.211019\n                      19.12.0.0.210720      not-available\n                      19.11.0.0.210420      not-available\n                      21.4.0.0.211019       not-available\n                      21.3.0.0.210720       not-available\n\nDB                    19.13.0.0.211019      19.13.0.0.211019\n                      19.12.0.0.210720      not-available\n                      19.11.0.0.210420      not-available\n                      21.4.0.0.211019       not-available\n                      21.3.0.0.210720       not-available\n<\/pre>\n
<\/br><\/p>\n
I have created the first DB System json file assigning a new IP address from the VLAN38 network using tagged38 virtual network. The IP connection will use bridging through the virtual VLAN38 IP address we created on the Bare Metal itself :<\/p>\n
\n[root@dbi-oda-x8 ~]# cat \/opt\/dbi\/create_dbsystem_srvdb38.json\n...\n...\n...\n\"network\": {\n    \"domainName\": \"dbi-lab.ch\",\n    \"ntpServers\": [\"216.239.35.0\"],\n    \"dnsServers\": [\n        \"8.8.8.8\",\"8.8.4.4\"\n    ],\n    \"nodes\": [\n        {\n            \"name\": \"srvdb38\",\n            \"ipAddress\": \"10.38.0.20\",\n            \"netmask\": \"255.255.255.0\",\n            \"gateway\": \"10.38.0.1\",\n            \"number\": 0\n        }\n    ],\n\"publicVNetwork\": \"tagged38\"\n},\n\"grid\": {\n    \"language\": \"en\"\n}\n}\n<\/pre>\n
<\/br><\/p>\n
Creation of the first DB System on network VLAN 38 :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-dbsystem -p \/opt\/dbi\/create_dbsystem_srvdb38.json\nEnter password for system \"srvdb38\":\nRetype password for system \"srvdb38\":\nEnter administrator password for DB \"DB38\":\nRetype administrator password for DB \"DB38\":\n\nJob details\n----------------------------------------------------------------\n                     ID:  ed88ef81-5cb3-4214-ac5c-bc255b67577f\n            Description:  DB System srvdb38 creation\n                 Status:  Created\n                Created:  February 4, 2022 4:12:33 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i ed88ef81-5cb3-4214-ac5c-bc255b67577f\n\nJob details\n----------------------------------------------------------------\n                     ID:  ed88ef81-5cb3-4214-ac5c-bc255b67577f\n            Description:  DB System srvdb38 creation\n                 Status:  Success\n                Created:  February 4, 2022 4:12:33 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nCreate DB System metadata                February 4, 2022 4:12:33 PM CET     February 4, 2022 4:12:33 PM CET     Success\nPersist new DB System                    February 4, 2022 4:12:33 PM CET     February 4, 2022 4:12:33 PM CET     Success\nValidate DB System prerequisites         February 4, 2022 4:12:33 PM CET     February 4, 2022 4:12:37 PM CET     Success\nSetup DB System environment              February 4, 2022 4:12:37 PM CET     February 4, 2022 4:12:39 PM CET     Success\nCreate DB System ASM volume              February 4, 2022 4:12:39 PM CET     February 4, 2022 4:12:45 PM CET     Success\nCreate DB System ACFS filesystem         February 4, 2022 4:12:45 PM CET     February 4, 2022 4:12:54 PM CET     Success\nCreate DB System VM ACFS snapshots       February 4, 2022 4:12:54 PM CET     February 4, 2022 4:13:27 PM CET     Success\nCreate temporary SSH key pair            February 4, 2022 4:13:27 PM CET     February 4, 2022 4:13:27 PM CET     Success\nCreate DB System cloud-init config       February 4, 2022 4:13:27 PM CET     February 4, 2022 4:13:27 PM CET     Success\nProvision DB System VM(s)                February 4, 2022 4:13:27 PM CET     February 4, 2022 4:13:28 PM CET     Success\nAttach disks to DB System                February 4, 2022 4:13:28 PM CET     February 4, 2022 4:13:29 PM CET     Success\nAdd DB System to Clusterware             February 4, 2022 4:13:29 PM CET     February 4, 2022 4:13:29 PM CET     Success\nStart DB System                          February 4, 2022 4:13:29 PM CET     February 4, 2022 4:13:30 PM CET     Success\nWait DB System VM first boot             February 4, 2022 4:13:30 PM CET     February 4, 2022 4:17:06 PM CET     Success\nSetup Mutual TLS (mTLS)                  February 4, 2022 4:17:06 PM CET     February 4, 2022 4:21:36 PM CET     Success\nExport clones repository                 February 4, 2022 4:21:36 PM CET     February 4, 2022 4:21:37 PM CET     Success\nSetup ASM client cluster config          February 4, 2022 4:21:37 PM CET     February 4, 2022 4:22:00 PM CET     Success\nInstall DB System                        February 4, 2022 4:22:00 PM CET     February 4, 2022 5:07:11 PM CET     Success\nCleanup temporary SSH key pair           February 4, 2022 5:07:11 PM CET     February 4, 2022 5:07:52 PM CET     Success\nSet DB System as configured              February 4, 2022 5:07:52 PM CET     February 4, 2022 5:07:52 PM CET     Success\n<\/pre>\n
<\/br><\/p>\n
I have then created the second DB System json file assigning a new IP address from the VLAN39 network using tagged39 virtual network. That time the IP connection will use bridging through the virtual VLAN39 IP address we created on the Bare Metal itself : <\/p>\n
\n[root@dbi-oda-x8 ~]# cat \/opt\/dbi\/create_dbsystem_srvdb39.json\n...\n...\n...\n},\n\"network\": {\n    \"domainName\": \"dbi-lab.ch\",\n    \"ntpServers\": [\"216.239.35.0\"],\n    \"dnsServers\": [\n        \"8.8.8.8\",\"8.8.4.4\"\n    ],\n    \"nodes\": [\n        {\n            \"name\": \"srvdb39\",\n            \"ipAddress\": \"10.39.0.20\",\n            \"netmask\": \"255.255.255.0\",\n            \"gateway\": \"10.39.0.1\",\n            \"number\": 0\n        }\n    ],\n\"publicVNetwork\": \"tagged39\"\n},\n\"grid\": {\n    \"language\": \"en\"\n}\n}\n<\/pre>\n
<\/br><\/p>\n
Creation on the second DB System on network VLAN 39 :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli create-dbsystem -p \/opt\/dbi\/create_dbsystem_srvdb39.json\nEnter password for system \"srvdb39\":\nRetype password for system \"srvdb39\":\nEnter administrator password for DB \"DB39\":\nRetype administrator password for DB \"DB39\":\n\nJob details\n----------------------------------------------------------------\n                     ID:  38303453-8524-4c65-b1d2-3717dfc79a1f\n            Description:  DB System srvdb39 creation\n                 Status:  Created\n                Created:  February 4, 2022 5:14:01 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i 38303453-8524-4c65-b1d2-3717dfc79a1f\n\nJob details\n----------------------------------------------------------------\n                     ID:  38303453-8524-4c65-b1d2-3717dfc79a1f\n            Description:  DB System srvdb39 creation\n                 Status:  Success\n                Created:  February 4, 2022 5:14:01 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nCreate DB System metadata                February 4, 2022 5:14:01 PM CET     February 4, 2022 5:14:01 PM CET     Success\nPersist new DB System                    February 4, 2022 5:14:01 PM CET     February 4, 2022 5:14:01 PM CET     Success\nValidate DB System prerequisites         February 4, 2022 5:14:01 PM CET     February 4, 2022 5:14:05 PM CET     Success\nSetup DB System environment              February 4, 2022 5:14:05 PM CET     February 4, 2022 5:14:07 PM CET     Success\nCreate DB System ASM volume              February 4, 2022 5:14:07 PM CET     February 4, 2022 5:14:13 PM CET     Success\nCreate DB System ACFS filesystem         February 4, 2022 5:14:13 PM CET     February 4, 2022 5:14:22 PM CET     Success\nCreate DB System VM ACFS snapshots       February 4, 2022 5:14:22 PM CET     February 4, 2022 5:14:52 PM CET     Success\nCreate temporary SSH key pair            February 4, 2022 5:14:52 PM CET     February 4, 2022 5:14:52 PM CET     Success\nCreate DB System cloud-init config       February 4, 2022 5:14:52 PM CET     February 4, 2022 5:14:53 PM CET     Success\nProvision DB System VM(s)                February 4, 2022 5:14:53 PM CET     February 4, 2022 5:14:54 PM CET     Success\nAttach disks to DB System                February 4, 2022 5:14:54 PM CET     February 4, 2022 5:14:54 PM CET     Success\nAdd DB System to Clusterware             February 4, 2022 5:14:54 PM CET     February 4, 2022 5:14:54 PM CET     Success\nStart DB System                          February 4, 2022 5:14:54 PM CET     February 4, 2022 5:14:56 PM CET     Success\nWait DB System VM first boot             February 4, 2022 5:14:56 PM CET     February 4, 2022 5:18:32 PM CET     Success\nSetup Mutual TLS (mTLS)                  February 4, 2022 5:18:32 PM CET     February 4, 2022 5:23:07 PM CET     Success\nExport clones repository                 February 4, 2022 5:23:07 PM CET     February 4, 2022 5:23:07 PM CET     Success\nSetup ASM client cluster config          February 4, 2022 5:23:07 PM CET     February 4, 2022 5:23:30 PM CET     Success\nInstall DB System                        February 4, 2022 5:23:30 PM CET     February 4, 2022 6:09:43 PM CET     Success\nCleanup temporary SSH key pair           February 4, 2022 6:09:43 PM CET     February 4, 2022 6:10:23 PM CET     Success\nSet DB System as configured              February 4, 2022 6:10:23 PM CET     February 4, 2022 6:10:23 PM CET     Success\n<\/pre>\n
<\/br><\/p>\n
Virtual network configuration checks<\/h3>\n
So I have got my both virtual networks configured on respective IP address (10.38.0.10 for VLAN38 network and 10.39.0.10 for VLAN39 network) :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\ntagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-04 15:19:45 CET  2022-02-04 15:19:45 CET\ntagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\n\n[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged38\nVNetwork details\n--------------------------------------------------------------------------------\n                       ID:  0d8099ba-ff42-4d73-82b4-f7497fb68ca5\n                     Name:  tagged38\n                  Created:  2022-02-04 15:19:07 CET\n                  Updated:  2022-02-04 15:19:07 CET\n                     Type:  BridgedVlan\n           Interface name:  btbond2\n              Bridge name:  brtagged38\n                  VLAN ID:  38\n                       IP:  10.38.0.10\n                  Netmask:  255.255.255.0\n                  Gateway:  10.38.0.1\n Attached in VMs (config):  xa287c764d\n   Attached in VMs (live):  xa287c764d\n\n[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged39\nVNetwork details\n--------------------------------------------------------------------------------\n                       ID:  00df04d4-9a01-4547-9a3c-5a27dab10494\n                     Name:  tagged39\n                  Created:  2022-02-04 15:19:45 CET\n                  Updated:  2022-02-04 15:19:45 CET\n                     Type:  BridgedVlan\n           Interface name:  btbond2\n              Bridge name:  brtagged39\n                  VLAN ID:  39\n                       IP:  10.39.0.10\n                  Netmask:  255.255.255.0\n                  Gateway:  10.39.0.1\n Attached in VMs (config):  x793d6c5ce\n   Attached in VMs (live):  x793d6c5ce\n<\/pre>\n
<\/br><\/p>\n
DB System information checks<\/h3>\n
I have my 2 DB Systems.
\nsrvdb38 configured on VLAN38 network with 10.38.0.20 IP Address.
\nsrvdb39 configured on VLAN39 network with 10.39.0.20 IP Address.<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-dbsystems\nName                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated\n--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------\nsrvdb39               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 17:14:01 CET  2022-02-04 18:10:23 CET\nsrvdb38               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 16:12:33 CET  2022-02-04 17:07:52 CET\n\n[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb38\nDB System details\n--------------------------------------------------------------------------------\n                       ID:  859466fe-a4a5-40ae-aa00-8e02cf74dedc\n                     Name:  srvdb38\n                    Image:  19.13.0.0.0\n                    Shape:  odb2\n             Cluster name:  dbsa287c764d\n             Grid version:  19.13.0.0.211019\n                   Memory:  16.00 GB\n             NUMA enabled:  YES\n                   Status:  CONFIGURED\n                  Created:  2022-02-04 16:12:33 CET\n                  Updated:  2022-02-04 17:07:52 CET\n\n CPU Pool\n--------------------------\n                     Name:  cpupool4dbsystems\n          Number of cores:  4\n\n                     Host:  dbi-oda-x8\n        Effective CPU set:  5-6,21-22,37-38,53-54\n              Online CPUs:  5, 6, 21, 22, 37, 38, 53, 54\n             Offline CPUs:  NONE\n\n VM Storage\n--------------------------\n               Disk group:  DATA\n              Volume name:  SA287C764D\n            Volume device:  \/dev\/asm\/sa287c764d-390\n                     Size:  200.00 GB\n              Mount Point:  \/u05\/app\/sharedrepo\/srvdb38\n\n VMs\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  xa287c764d\n             VM Host Name:  srvdb38.dbi-lab.ch\n            VM image path:  \/u05\/app\/sharedrepo\/srvdb38\/.ACFS\/snaps\/vm_xa287c764d\/xa287c764d\n             Target State:  ONLINE\n            Current State:  ONLINE\n\n VNetworks\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  xa287c764d\n                   Public:  10.38.0.20      \/ 255.255.255.0   \/ ens3 \/ BRIDGE(brtagged38)\n                      ASM:  192.168.17.4    \/ 255.255.255.128 \/ ens4 \/ BRIDGE(privasm) VLAN(priv0.100)\n\n Extra VNetworks\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  xa287c764d\n                 tagged38:  10.38.0.20      \/ 255.255.255.0   \/ PUBLIC\n\n Databases\n--------------------------\n                     Name:  DB38\n              Resource ID:  5ff22777-f24b-446a-8c58-3fa6dd2ad383\n              Unique name:  DB38_SITE1\n              Database ID:  715007309\n              Domain name:  dbi-lab.ch\n               DB Home ID:  8b66a7d6-9e5e-43c0-a69c-00b80d2dba81\n                    Shape:  odb2\n                  Version:  19.13.0.0.211019\n                  Edition:  EE\n                     Type:  SI\n                     Role:  PRIMARY\n                    Class:  OLTP\n                  Storage:  ASM\n               Redundancy:\n         Target node name:\n            Character set:  AL32UTF8\n        NLS character set:\n                 Language:  ENGLISH\n                Territory:  AMERICA\n          Console enabled:  false\n             SEHA enabled:  false\n      Associated networks:  Public-network\n         Backup config ID:\n       Level 0 Backup Day:  sunday\n       Autobackup enabled:  true\n              TDE enabled:  false\n                 CDB type:  false\n                 PDB name:\n           PDB admin user:\n\n[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb39\nDB System details\n--------------------------------------------------------------------------------\n                       ID:  5238f074-6126-4051-8b9e-e392b55a2328\n                     Name:  srvdb39\n                    Image:  19.13.0.0.0\n                    Shape:  odb2\n             Cluster name:  dbs793d6c5ce\n             Grid version:  19.13.0.0.211019\n                   Memory:  16.00 GB\n             NUMA enabled:  YES\n                   Status:  CONFIGURED\n                  Created:  2022-02-04 17:14:01 CET\n                  Updated:  2022-02-04 18:10:23 CET\n\n CPU Pool\n--------------------------\n                     Name:  cpupool4dbsystems\n          Number of cores:  4\n\n                     Host:  dbi-oda-x8\n        Effective CPU set:  5-6,21-22,37-38,53-54\n              Online CPUs:  5, 6, 21, 22, 37, 38, 53, 54\n             Offline CPUs:  NONE\n\n VM Storage\n--------------------------\n               Disk group:  DATA\n              Volume name:  S793D6C5CE\n            Volume device:  \/dev\/asm\/s793d6c5ce-390\n                     Size:  200.00 GB\n              Mount Point:  \/u05\/app\/sharedrepo\/srvdb39\n\n VMs\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  x793d6c5ce\n             VM Host Name:  srvdb39.dbi-lab.ch\n            VM image path:  \/u05\/app\/sharedrepo\/srvdb39\/.ACFS\/snaps\/vm_x793d6c5ce\/x793d6c5ce\n             Target State:  ONLINE\n            Current State:  ONLINE\n\n VNetworks\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  x793d6c5ce\n                   Public:  10.39.0.20      \/ 255.255.255.0   \/ ens3 \/ BRIDGE(brtagged39)\n                      ASM:  192.168.17.5    \/ 255.255.255.128 \/ ens4 \/ BRIDGE(privasm) VLAN(priv0.100)\n\n Extra VNetworks\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  x793d6c5ce\n                 tagged39:  10.39.0.20      \/ 255.255.255.0   \/ PUBLIC\n\n Databases\n--------------------------\n                     Name:  DB39\n              Resource ID:  e796e7e1-c8c8-4a63-809c-57976ce2163d\n              Unique name:  DB39_SITE1\n              Database ID:  2043690133\n              Domain name:  dbi-lab.ch\n               DB Home ID:  67f6df00-83d8-477b-8884-201664a3701b\n                    Shape:  odb2\n                  Version:  19.13.0.0.211019\n                  Edition:  EE\n                     Type:  SI\n                     Role:  PRIMARY\n                    Class:  OLTP\n                  Storage:  ASM\n               Redundancy:\n         Target node name:\n            Character set:  AL32UTF8\n        NLS character set:\n                 Language:  ENGLISH\n                Territory:  AMERICA\n          Console enabled:  false\n             SEHA enabled:  false\n      Associated networks:  Public-network\n         Backup config ID:\n       Level 0 Backup Day:  sunday\n       Autobackup enabled:  true\n              TDE enabled:  false\n                 CDB type:  false\n                 PDB name:\n           PDB admin user:\n\n[root@dbi-oda-x8 ~]#\n<\/pre>\n
<\/br><\/p>\n
Direct connection on the DB Systems<\/h3>\n
Finally I can access directly the DB Systems through SSH from my laptop. Here as example, connection to the srvdb38 DB System.<\/p>\n
\nmaw@DBI-LT-MAW2 ~ % ssh root@10.38.0.20\nroot@10.38.0.20's password:\nLast login: Thu Feb 17 16:07:31 2022\n\n[root@srvdb38 ~]# ip addr sh ens3\n2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n    link\/ether 52:54:00:6e:3f:9f brd ff:ff:ff:ff:ff:ff\n    inet 10.38.0.20\/24 brd 10.38.0.255 scope global noprefixroute ens3\n       valid_lft forever preferred_lft forever\n       \n[root@srvdb38 ~]# ps -ef | grep [p]mon\noracle   21506     1  0 14:05 ?        00:00:00 ora_pmon_DB38\n[root@srvdb38 ~]#\n<\/pre>\n
<\/br><\/p>\n
And directly from my laptop I can connect and use the Oracle database DB38 :
\n\"\"<\/a>
\n<\/br>
\n<\/br>
\n<\/br>
\n<\/br><\/p>\n
Creating a virtual network without assigning an IP address to the bridge interface<\/h3>\n
Of course it is possible to create a virtual bridge interface without assigning any IP address.<\/p>\n
I deleted tagged39 virtual network in order to create it again without IP address :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\ntagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\ntagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-04 22:13:18 CET  2022-02-17 16:38:02 CET\n\n[root@dbi-oda-x8 ~]# odacli delete-vnetwork -n tagged39\n\nJob details\n----------------------------------------------------------------\n                     ID:  2cae44c4-39e7-4102-8453-597f707ed10b\n            Description:  vNetwork tagged39 deletion\n                 Status:  Created\n                Created:  February 17, 2022 4:46:45 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i 2cae44c4-39e7-4102-8453-597f707ed10b\n\nJob details\n----------------------------------------------------------------\n                     ID:  2cae44c4-39e7-4102-8453-597f707ed10b\n            Description:  vNetwork tagged39 deletion\n                 Status:  Success\n                Created:  February 17, 2022 4:46:45 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nValidate Virtual Network exists          February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:45 PM CET    Success\nValidate VM does not have the VNetwork attached February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:45 PM CET    Success\nDelete bridge                            February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:45 PM CET    Success\nDelete VLAN                              February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:46 PM CET    Success\nDelete metadata                          February 17, 2022 4:46:46 PM CET    February 17, 2022 4:46:46 PM CET    Success\n\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\ntagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\n\n[root@dbi-oda-x8 ~]# ip addr sh | grep btbond2\n4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n10: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n11: btbond2.38@btbond2:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000\n[root@dbi-oda-x8 ~]#\n<\/pre>\n
<\/br><\/p>\n
Of course I could have updated the vnetwork interface using odacli modify-vnetwork and assigning 0.0.0.0 as IP address.<\/p>\n
I created again the tagged39 virtual network without assigning an IP address to the bridge interface.<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\ntagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\n\n[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -vlan 39\n\nJob details\n----------------------------------------------------------------\n                     ID:  a70a1313-f2cd-48ab-ac5f-c17b2f2fae08\n            Description:  vNetwork tagged39 creation\n                 Status:  Created\n                Created:  February 17, 2022 4:49:44 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i a70a1313-f2cd-48ab-ac5f-c17b2f2fae08\n\nJob details\n----------------------------------------------------------------\n                     ID:  a70a1313-f2cd-48ab-ac5f-c17b2f2fae08\n            Description:  vNetwork tagged39 creation\n                 Status:  Success\n                Created:  February 17, 2022 4:49:44 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nValidate Virtual Network doesn't exist   February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success\nValidate interface to use exists         February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success\nValidate interfaces to create not exist  February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success\nCreate bridge                            February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success\nCreate VLAN                              February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success\nBring up VLAN                            February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:45 PM CET    Success\nCreate metadata                          February 17, 2022 4:49:45 PM CET    February 17, 2022 4:49:45 PM CET    Success\nPersist metadata                         February 17, 2022 4:49:45 PM CET    February 17, 2022 4:49:45 PM CET    Success\n\n[root@dbi-oda-x8 ~]# odacli list-vnetworks\nName                  Type             Interface        Bridge                Uniform   Created                  Updated\n--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------\ntagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET\npubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET\ntagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-17 16:49:45 CET  2022-02-17 16:49:45 CET\n\n[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged39\nVNetwork details\n--------------------------------------------------------------------------------\n                       ID:  6e40ca21-1d4a-4bcc-8436-f29ffb2eb8bc\n                     Name:  tagged39\n                  Created:  2022-02-17 16:49:45 CET\n                  Updated:  2022-02-17 16:49:45 CET\n                     Type:  BridgedVlan\n           Interface name:  btbond2\n              Bridge name:  brtagged39\n                  VLAN ID:  39\n                       IP:\n                  Netmask:\n                  Gateway:\n Attached in VMs (config):  NONE\n   Attached in VMs (live):  NONE\n\n[root@dbi-oda-x8 ~]# ip addr sh | grep btbond2\n4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000\n10: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n11: btbond2.38@btbond2:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000\n31: btbond2.39@btbond2:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000\n\n[root@dbi-oda-x8 ~]# ip addr sh | grep brtagged\n11: btbond2.38@btbond2:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000\n12: brtagged38:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n    inet 10.38.0.10\/24 brd 10.38.0.255 scope global brtagged38\n25: vnet6:  mtu 1500 qdisc pfifo_fast master brtagged38 state UNKNOWN group default qlen 1000\n30: brtagged39:  mtu 1500 qdisc noqueue state UP group default qlen 1000\n31: btbond2.39@btbond2:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000\n[root@dbi-oda-x8 ~]#\n<\/pre>\n
<\/br><\/p>\n
And let’s deploy the srvdb39 DB Systems again after having removed it with odacli delete-dbsystem command :<\/p>\n
\n[root@dbi-oda-x8 ~]# odacli list-dbsystems\nName                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated\n--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------\nsrvdb38               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 16:12:33 CET  2022-02-04 17:07:52 CET\n\n[root@dbi-oda-x8 ~]# odacli create-dbsystem -p \/opt\/dbi\/create_dbsystem_srvdb39.json\nEnter password for system \"srvdb39\":\nRetype password for system \"srvdb39\":\nEnter administrator password for DB \"DB39\":\nRetype administrator password for DB \"DB39\":\n\nJob details\n----------------------------------------------------------------\n                     ID:  934c752b-7eb6-4b5b-8856-7a90102a1e65\n            Description:  DB System srvdb39 creation\n                 Status:  Created\n                Created:  February 17, 2022 5:06:37 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@dbi-oda-x8 ~]# odacli describe-job -i 934c752b-7eb6-4b5b-8856-7a90102a1e65\n\nJob details\n----------------------------------------------------------------\n                     ID:  934c752b-7eb6-4b5b-8856-7a90102a1e65\n            Description:  DB System srvdb39 creation\n                 Status:  Success\n                Created:  February 17, 2022 5:06:37 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nCreate DB System metadata                February 17, 2022 5:06:37 PM CET    February 17, 2022 5:06:37 PM CET    Success\nPersist new DB System                    February 17, 2022 5:06:37 PM CET    February 17, 2022 5:06:37 PM CET    Success\nValidate DB System prerequisites         February 17, 2022 5:06:37 PM CET    February 17, 2022 5:06:41 PM CET    Success\nSetup DB System environment              February 17, 2022 5:06:41 PM CET    February 17, 2022 5:06:43 PM CET    Success\nCreate DB System ASM volume              February 17, 2022 5:06:43 PM CET    February 17, 2022 5:06:49 PM CET    Success\nCreate DB System ACFS filesystem         February 17, 2022 5:06:49 PM CET    February 17, 2022 5:06:58 PM CET    Success\nCreate DB System VM ACFS snapshots       February 17, 2022 5:06:58 PM CET    February 17, 2022 5:07:31 PM CET    Success\nCreate temporary SSH key pair            February 17, 2022 5:07:31 PM CET    February 17, 2022 5:07:32 PM CET    Success\nCreate DB System cloud-init config       February 17, 2022 5:07:32 PM CET    February 17, 2022 5:07:32 PM CET    Success\nProvision DB System VM(s)                February 17, 2022 5:07:32 PM CET    February 17, 2022 5:07:33 PM CET    Success\nAttach disks to DB System                February 17, 2022 5:07:33 PM CET    February 17, 2022 5:07:34 PM CET    Success\nAdd DB System to Clusterware             February 17, 2022 5:07:34 PM CET    February 17, 2022 5:07:34 PM CET    Success\nStart DB System                          February 17, 2022 5:07:34 PM CET    February 17, 2022 5:07:36 PM CET    Success\nWait DB System VM first boot             February 17, 2022 5:07:36 PM CET    February 17, 2022 5:08:48 PM CET    Success\nSetup Mutual TLS (mTLS)                  February 17, 2022 5:08:48 PM CET    February 17, 2022 5:09:08 PM CET    Success\nExport clones repository                 February 17, 2022 5:09:08 PM CET    February 17, 2022 5:09:08 PM CET    Success\nSetup ASM client cluster config          February 17, 2022 5:09:08 PM CET    February 17, 2022 5:09:11 PM CET    Success\nInstall DB System                        February 17, 2022 5:09:11 PM CET    February 17, 2022 5:36:57 PM CET    Success\nCleanup temporary SSH key pair           February 17, 2022 5:36:57 PM CET    February 17, 2022 5:36:57 PM CET    Success\nSet DB System as configured              February 17, 2022 5:36:57 PM CET    February 17, 2022 5:36:57 PM CET    Success\n\n[root@dbi-oda-x8 ~]# odacli list-dbsystems\nName                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated\n--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------\nsrvdb38               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 16:12:33 CET  2022-02-04 17:07:52 CET\nsrvdb39               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-17 17:06:37 CET  2022-02-17 17:36:57 CET\n\n[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb39\nDB System details\n--------------------------------------------------------------------------------\n                       ID:  ed13bacb-be28-4e31-a5d7-39fd713d72a0\n                     Name:  srvdb39\n                    Image:  19.13.0.0.0\n                    Shape:  odb2\n             Cluster name:  dbs793d6c5ce\n             Grid version:  19.13.0.0.211019\n                   Memory:  16.00 GB\n             NUMA enabled:  YES\n                   Status:  CONFIGURED\n                  Created:  2022-02-17 17:06:37 CET\n                  Updated:  2022-02-17 17:36:57 CET\n\n CPU Pool\n--------------------------\n                     Name:  cpupool4dbsystems\n          Number of cores:  4\n\n                     Host:  dbi-oda-x8\n        Effective CPU set:  5-6,21-22,37-38,53-54\n              Online CPUs:  5, 6, 21, 22, 37, 38, 53, 54\n             Offline CPUs:  NONE\n\n VM Storage\n--------------------------\n               Disk group:  DATA\n              Volume name:  S793D6C5CE\n            Volume device:  \/dev\/asm\/s793d6c5ce-390\n                     Size:  200.00 GB\n              Mount Point:  \/u05\/app\/sharedrepo\/srvdb39\n\n VMs\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  x793d6c5ce\n             VM Host Name:  srvdb39.dbi-lab.ch\n            VM image path:  \/u05\/app\/sharedrepo\/srvdb39\/.ACFS\/snaps\/vm_x793d6c5ce\/x793d6c5ce\n             Target State:  ONLINE\n            Current State:  ONLINE\n\n VNetworks\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  x793d6c5ce\n                   Public:  10.39.0.20      \/ 255.255.255.0   \/ ens3 \/ BRIDGE(brtagged39)\n                      ASM:  192.168.17.5    \/ 255.255.255.128 \/ ens4 \/ BRIDGE(privasm) VLAN(priv0.100)\n\n Extra VNetworks\n--------------------------\n                     Host:  dbi-oda-x8\n                  VM Name:  x793d6c5ce\n                 tagged39:  10.39.0.20      \/ 255.255.255.0   \/ PUBLIC\n\n Databases\n--------------------------\n                     Name:  DB39\n              Resource ID:  6ecee323-cd18-4971-acb4-b476d4d76d2c\n              Unique name:  DB39_SITE1\n              Database ID:  2044811647\n              Domain name:  dbi-lab.ch\n               DB Home ID:  3a6f8134-407a-4272-8512-7c52bdca68ea\n                    Shape:  odb2\n                  Version:  19.13.0.0.211019\n                  Edition:  EE\n                     Type:  SI\n                     Role:  PRIMARY\n                    Class:  OLTP\n                  Storage:  ASM\n               Redundancy:\n         Target node name:\n            Character set:  AL32UTF8\n        NLS character set:\n                 Language:  ENGLISH\n                Territory:  AMERICA\n          Console enabled:  false\n             SEHA enabled:  false\n      Associated networks:  Public-network\n         Backup config ID:\n       Level 0 Backup Day:  sunday\n       Autobackup enabled:  true\n              TDE enabled:  false\n                 CDB type:  false\n                 PDB name:\n           PDB admin user:\n<\/pre>\n
<\/br><\/p>\n
This DB System is also now accessible through his IP address 10.39.0.20 and is using the bridge interface without any IP address. A bridge interface does not need any IP address.<\/p>\n
I can ping the new DB System from my laptop :<\/p>\n
\nmaw@DBI-LT-MAW2 ~ % ping 10.39.0.20\nPING 10.39.0.20 (10.39.0.20): 56 data bytes\n64 bytes from 10.39.0.20: icmp_seq=0 ttl=62 time=56.761 ms\n64 bytes from 10.39.0.20: icmp_seq=1 ttl=62 time=56.734 ms\n64 bytes from 10.39.0.20: icmp_seq=2 ttl=62 time=64.759 ms\n^C\n--- 10.39.0.20 ping statistics ---\n3 packets transmitted, 3 packets received, 0.0% packet loss\nround-trip min\/avg\/max\/stddev = 56.734\/59.418\/64.759\/3.777 ms\n<\/pre>\n
<\/br><\/p>\n
I can ssh the new DB System from my laptop :<\/p>\n
\nmaw@DBI-LT-MAW2 ~ % ssh root@10.39.0.20\nroot@10.39.0.20's password:\nLast login: Thu Feb 17 18:10:10 2022 from gateway\n\n[root@srvdb39 ~]# ip addr sh ens3\n2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\n    link\/ether 52:54:00:86:87:88 brd ff:ff:ff:ff:ff:ff\n    inet 10.39.0.20\/24 brd 10.39.0.255 scope global noprefixroute ens3\n       valid_lft forever preferred_lft forever\n       \n[root@srvdb39 ~]# ps -ef | grep [p]mon\noracle   93401     1  0 17:35 ?        00:00:00 ora_pmon_DB39\n[root@srvdb39 ~]#\n<\/pre>\n
<\/br><\/p>\n
And I can join the DB39 database from my laptop directly :
\n\"\"<\/a>
\n<\/br>
\n<\/br>
\n<\/br>
\n<\/br>
\n\"\"<\/a>
\n<\/br>
\n<\/br>
\n<\/br>
\n<\/br>
\n<\/br>
\n<\/br><\/p>\n
Conclusion<\/h3>\n
KVM DB System main purpose is hard partitioning for oracle licensing. All ODA’s cpu cores can remain active. DB System with appropriate pinned CPU core related to the number of licenses will be created for the oracle database. The others ODA’s CPU core can be used for further KVM or application needs.
\nAs we could see KVM DB System will then have an additional pros. With DB System we can separate Database traffic through separated VLANs.<\/p>\n
On the other hand KVM DB System will have some other cons :<\/p>\n
    \n
  • More complex installation.<\/li>\n
  • More resources will be consumed : each DB Systems will run its own oracle restart (grid) infrastructure.<\/li>\n
  • Each DB Systems will have to be patched separately as well during ODA patching, making patching process more heavy.<\/li>\n<\/ul>\n
    Also please note that current ODA release 19.13 supports only creation of database in version 19.11, 19.12, 19.13, 21.3 and 21.4 for the DB Systems.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how […]<\/p>\n","protected":false},"author":48,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[59],"tags":[2400,2146,317,79,96,2471],"type_dbi":[],"class_list":["post-17103","post","type-post","status-publish","format-standard","hentry","category-oracle","tag-db-system","tag-kvm","tag-network","tag-oda","tag-oracle","tag-vlan"],"acf":[],"yoast_head":"\nCreating KVM Database System on separate VLAN network on ODA - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Creating KVM Database System on separate VLAN network on ODA\" \/>\n<meta property=\"og:description\" content=\"Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-02-04T19:25:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-05-31T14:33:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png\" \/>\n<meta name=\"author\" content=\"Marc Wagner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marc Wagner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"36 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\"},\"author\":{\"name\":\"Marc Wagner\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\"},\"headline\":\"Creating KVM Database System on separate VLAN network on ODA\",\"datePublished\":\"2022-02-04T19:25:46+00:00\",\"dateModified\":\"2022-05-31T14:33:44+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\"},\"wordCount\":1435,\"commentCount\":4,\"image\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png\",\"keywords\":[\"DB system\",\"kvm\",\"Network\",\"ODA\",\"Oracle\",\"VLAN\"],\"articleSection\":[\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\",\"name\":\"Creating KVM Database System on separate VLAN network on ODA - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png\",\"datePublished\":\"2022-02-04T19:25:46+00:00\",\"dateModified\":\"2022-05-31T14:33:44+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png\",\"contentUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Creating KVM Database System on separate VLAN network on ODA\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\",\"name\":\"Marc Wagner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"caption\":\"Marc Wagner\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/marc-wagner\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Creating KVM Database System on separate VLAN network on ODA - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/","og_locale":"en_US","og_type":"article","og_title":"Creating KVM Database System on separate VLAN network on ODA","og_description":"Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how […]","og_url":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/","og_site_name":"dbi Blog","article_published_time":"2022-02-04T19:25:46+00:00","article_modified_time":"2022-05-31T14:33:44+00:00","og_image":[{"url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png","type":"","width":"","height":""}],"author":"Marc Wagner","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Marc Wagner","Est. reading time":"36 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/"},"author":{"name":"Marc Wagner","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628"},"headline":"Creating KVM Database System on separate VLAN network on ODA","datePublished":"2022-02-04T19:25:46+00:00","dateModified":"2022-05-31T14:33:44+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/"},"wordCount":1435,"commentCount":4,"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png","keywords":["DB system","kvm","Network","ODA","Oracle","VLAN"],"articleSection":["Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/","url":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/","name":"Creating KVM Database System on separate VLAN network on ODA - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png","datePublished":"2022-02-04T19:25:46+00:00","dateModified":"2022-05-31T14:33:44+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#primaryimage","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/02\/Screenshot-2022-02-17-at-16.18.55.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/creating-kvm-database-system-on-separate-vlan-network-on-oda\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Creating KVM Database System on separate VLAN network on ODA"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628","name":"Marc Wagner","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","caption":"Marc Wagner"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/marc-wagner\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/17103","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=17103"}],"version-history":[{"count":1,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/17103\/revisions"}],"predecessor-version":[{"id":17104,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/17103\/revisions\/17104"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=17103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=17103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=17103"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=17103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}