Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how to create additional network on the ODA and how to create DB System on separate VLAN interface. The principle would be of course the same if we would like to create KVM Virtual Machines (compute instance) on separate VLAN.

Configuration description

For my test I have got an ODA X8-2M with a quad-port 10GBase-T network interface. I’m running ODA version 19.13.
On my network card, the 2 first ports p7p1 and p7p2 will be assigned to btbond1 and the 2 next ports p7p3 and p7p4 will be assigned to my second bonding interface btbond2. ODA are configured by default with active-backup mode without LACP for all bonding interfaces. This is automatically designed by the ODA and can not be changed. More over we need to keep in mind that on an appliance we will never manually change the Linux network scripts. All network configuration changes needs to be done with odacli.

btbond1 is used for my main network and we will use btbond2 to add additionnal networks.

The 2 additional networks are :
10.38.0.1/24 VLAN id 38
10.39.0.1/24 VLAN id 39

Checking network interface

With ethtool I can check and see that the 2 p7p3 and p7p4 ports are twisted pair and connected to the network :

[[email protected] ~]# ethtool p7p3 | grep -iE "(bportb|detected)"
	Port: Twisted Pair
	Link detected: yes

[[email protected] ~]# ethtool p7p4 | grep -iE "(bportb|detected)"
	Port: Twisted Pair
	Link detected: yes


I can see that both those interfaces are assigned to btbond2 interface :

[[email protected] ~]# grep MASTER /etc/sysconfig/network-scripts/ifcfg-p7p3
MASTER=btbond2

[[email protected] ~]# grep MASTER /etc/sysconfig/network-scripts/ifcfg-p7p4
MASTER=btbond2


Checking existing and default ODA configuration

After reimaging an ODA we would have the below default configuration.

[email protected] ~]# odacli list-networks

ID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET

[[email protected] ~]#


To use a network interface with the VMs, either a DB System or a Compute instance, the network interface needs to be created in the vnetworks.

VLAN Tagged versus Untagged

If a port on a switch is configured as tagged (trunk port in cisco world), the connected equipment (here our ODA) is VLAN aware, and would need to be tagged as well. The port is enabled for VLAN tagging. Purpose is to pass traffic for multiple VLAN’s. The ethernet frame is enclosing an additional VLAN header. The connected equipment will add the VLAN information in the ethernet frame.

If a port is configured as untagged (access port in cisco world), the connected equipment (here our ODA) does not know anything about in which VLAN it is and does not care about it. The switch will manage it on its own. The port does not tag and only accepts a single VLAN. The ethernet frame coming to the connected equipment will not have any VLAN header.

This is explained in the 802.1Q standard.

Most of the time trunk ports will link switches and access ports will link to end devices, albeit we would like to use several VLAN network on the same ODA network interface.

Creating an additional untagged network

Let’s create an additional untagged physical network on the Bare Metal itself. We will create it on btbond2 interface. Untagged means that the ODA will not add VLAN information in the Ethernet frame.
The option -t bond would be the default one and would here not be required.

[[email protected] ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m untagged1 -s 255.255.255.0 -g 10.38.0.1 -t bond
{
  "jobId" : "9a4476dd-b955-433b-9463-377f66ab737a",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "February 04, 2022 13:48:12 PM CET",
  "resourceList" : [ ],
  "description" : "Rac Network service creation with name untagged1",
  "updatedTime" : "February 04, 2022 13:48:12 PM CET"
}

[[email protected] ~]# odacli describe-job -i 9a4476dd-b955-433b-9463-377f66ab737a

Job details
----------------------------------------------------------------
                     ID:  9a4476dd-b955-433b-9463-377f66ab737a
            Description:  Rac Network service creation with name untagged1
                 Status:  Success
                Created:  February 4, 2022 1:48:12 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting network                          February 4, 2022 1:48:12 PM CET     February 4, 2022 1:48:18 PM CET     Success
Setting up Network                       February 4, 2022 1:48:12 PM CET     February 4, 2022 1:48:12 PM CET     Success
restart network interface btbond2        February 4, 2022 1:48:12 PM CET     February 4, 2022 1:48:18 PM CET     Success

[[email protected] ~]# odacli list-networks

ID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]
6171aad7-e247-4b08-a56e-83bdedd74af1   untagged1            btbond2      BOND            255.255.255.0      10.38.0.1                   [IP Address on node0: 10.38.0.10]

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET


I can see that our new network exists in the network list but not in the virtual network of course.

I can also check and see that our btbond2 interface has been configured by odacli as an untagged bonding interface with the appropriate information.

[[email protected] ~]# ip addr sh btbond2
9: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:fd:fe:92:80:1a brd ff:ff:ff:ff:ff:ff
    inet 10.38.0.10/24 brd 10.38.0.255 scope global btbond2
       valid_lft forever preferred_lft forever

[[email protected] ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*
-rw-r--r--. 1 root root 262 Feb  4 13:48 /etc/sysconfig/network-scripts/ifcfg-btbond2

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2
#This file was created by ODA. Do not edit.
NETMASK=255.255.255.0
GATEWAY=10.38.0.1
BOOTPROTO=none
PEERDNS=no
DEVICE=btbond2
ONBOOT=yes
NM_CONTROLLED=no
IPADDR=10.38.0.10
BONDING_OPTS="mode=active-backup miimon=100 primary=p7p3"
IPV6INIT=no
USERCTL=no
TYPE=BOND


Of course it is not possible to create another untagged network on btbond2 interface knowing there is already one existing :

[[email protected] ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bond
DCS-10001:Internal error encountered: nicnamebtbond2 already exists in the networks list .. .


Creating an additional tagged network would also not be possible. Note the option -t and -v to create tagged networks on the ODA :

[[email protected] ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged2 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39
DCS-10001:Internal error encountered: Creating vlan in the interface btbond2 is not allowed. Physical network untagged1 already exists in interface btbond2.


Let’s delete our additional untagged network

[[email protected] ~]# odacli delete-network -m untagged1
{
  "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
  "status" : "Running",
  "message" : null,
  "reports" : [ {
    "taskId" : "TaskSequential_10041",
    "taskName" : "deleting network",
    "taskResult" : "",
    "startTime" : "February 04, 2022 14:08:32 PM CET",
    "endTime" : "February 04, 2022 14:08:32 PM CET",
    "status" : "Running",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_10039",
    "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "February 04, 2022 14:08:32 PM CET"
  }, {
    "taskId" : "TaskZJsonRpcExt_10045",
    "taskName" : "Setting up Network",
    "taskResult" : "Network setup success",
    "startTime" : "February 04, 2022 14:08:32 PM CET",
    "endTime" : "February 04, 2022 14:08:32 PM CET",
    "status" : "Success",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_10041",
    "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "February 04, 2022 14:08:32 PM CET"
  }, {
    "taskId" : "TaskZJsonRpcExt_10048",
    "taskName" : "restart network interface btbond2",
    "taskResult" : "",
    "startTime" : "February 04, 2022 14:08:32 PM CET",
    "endTime" : "February 04, 2022 14:08:32 PM CET",
    "status" : "Running",
    "taskDescription" : null,
    "parentTaskId" : "TaskSequential_10047",
    "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
    "tags" : [ ],
    "reportLevel" : "Info",
    "updatedTime" : "February 04, 2022 14:08:32 PM CET"
  } ],
  "createTimestamp" : "February 04, 2022 14:08:32 PM CET",
  "resourceList" : [ {
    "resourceId" : "6171aad7-e247-4b08-a56e-83bdedd74af1",
    "resourceType" : null,
    "resourceNewType" : "Network",
    "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
    "updatedTime" : null
  } ],
  "description" : "Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1",
  "updatedTime" : "February 04, 2022 14:08:32 PM CET"
}

[[email protected] ~]# odacli describe-job -i 3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57

Job details
----------------------------------------------------------------
                     ID:  3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57
            Description:  Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1
                 Status:  Success
                Created:  February 4, 2022 2:08:32 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
deleting network                         February 4, 2022 2:08:32 PM CET     February 4, 2022 2:08:34 PM CET     Success
Setting up Network                       February 4, 2022 2:08:32 PM CET     February 4, 2022 2:08:32 PM CET     Success
restart network interface btbond2        February 4, 2022 2:08:32 PM CET     February 4, 2022 2:08:34 PM CET     Success


Network has been deleted :

[[email protected] ~]# odacli list-networks

ID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]


Creating additional tagged network

Let’s create the first tagged network on btbond2 interface :

[[email protected] ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m tagged38 -s 255.255.255.0 -g 10.38.0.1 -t VLAN -v 38
{
  "jobId" : "b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "February 04, 2022 14:40:28 PM CET",
  "resourceList" : [ ],
  "description" : "Rac Network service creation with name tagged38",
  "updatedTime" : "February 04, 2022 14:40:28 PM CET"
}

[[email protected] ~]# odacli describe-job -i b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc

Job details
----------------------------------------------------------------
                     ID:  b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc
            Description:  Rac Network service creation with name tagged38
                 Status:  Success
                Created:  February 4, 2022 2:40:28 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting network                          February 4, 2022 2:40:28 PM CET     February 4, 2022 2:40:33 PM CET     Success
Setting up Vlan                          February 4, 2022 2:40:28 PM CET     February 4, 2022 2:40:33 PM CET     Success


The tagged network has been created :

[[email protected] ~]# odacli list-networks

ID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]
fbe05bfa-636e-4f9c-a348-c59ba23e2296   tagged38             btbond2.38   VLAN            255.255.255.0      10.38.0.1          38       [IP Address on node0: 10.38.0.10]


And we can see that respective tagged network interface has now been created on the Linux operating system side (note the <btbond>.<vlan_id>) :

[[email protected] ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*
-rw-r--r--. 1 root root 239 Feb  4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2
-rw-r--r--  1 root root 520 Feb  4 14:40 /etc/sysconfig/network-scripts/ifcfg-btbond2.38

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.38
#ODA_VLAN_CONFIG ===
#ODA_VLAN_CONFIG Name=tagged38
#ODA_VLAN_CONFIG VlanId=38
#ODA_VLAN_CONFIG VlanInterface=btbond2
#ODA_VLAN_CONFIG Type=VlanType
#ODA_VLAN_CONFIG VlanSetupType=Other
#ODA_VLAN_CONFIG VlanIpAddr=10.38.0.10
#ODA_VLAN_CONFIG VlanNetmask=255.255.255.0
#ODA_VLAN_CONFIG VlanGateway=10.38.0.1
#ODA_VLAN_CONFIG NodeNum=0
#=== DO NOT EDIT ANYTHING ABOVE THIS LINE ===
DEVICE=btbond2.38
BOOTPROTO=none
ONBOOT=yes
VLAN=yes
NM_CONTROLLED=no
DEFROUTE=no
IPADDR=10.38.0.10
NETMASK=255.255.255.0
GATEWAY=10.38.0.1
[[email protected] ~]#


And I can reach the ODA on the new network :

C:Users>ping 10.38.0.10

Pinging 10.38.0.10 with 32 bytes of data:
Reply from 10.38.0.10: bytes=32 time=1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63

Ping statistics for 10.38.0.10:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms


The ODA will not permit to create any untagged additional network on the btbond2 interface knowing one tagged network already exists :

[[email protected] ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bond
DCS-10001:Internal error encountered: Creating non-VLAN typed network on the interface btbond2 is not allowed. VLAN tagged38 already exists in interface btbond2.


Let’s create the second tagged network on the same btbond2 interface :

[[email protected] ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged39 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39
{
  "jobId" : "e0452663-544c-47c4-8b9b-5fbe6e0a5cd9",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "February 04, 2022 14:45:18 PM CET",
  "resourceList" : [ ],
  "description" : "Rac Network service creation with name tagged39",
  "updatedTime" : "February 04, 2022 14:45:18 PM CET"
}

[[email protected] ~]# odacli describe-job -i e0452663-544c-47c4-8b9b-5fbe6e0a5cd9

Job details
----------------------------------------------------------------
                     ID:  e0452663-544c-47c4-8b9b-5fbe6e0a5cd9
            Description:  Rac Network service creation with name tagged39
                 Status:  Success
                Created:  February 4, 2022 2:45:18 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting network                          February 4, 2022 2:45:18 PM CET     February 4, 2022 2:45:23 PM CET     Success
Setting up Vlan                          February 4, 2022 2:45:18 PM CET     February 4, 2022 2:45:23 PM CET     Success


New tagged network has been created and we now have 2 tagged network running on the same btbond2 interface :

[[email protected] ~]# odacli list-networks

ID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]
fbe05bfa-636e-4f9c-a348-c59ba23e2296   tagged38             btbond2.38   VLAN            255.255.255.0      10.38.0.1          38       [IP Address on node0: 10.38.0.10]
dc11ecbe-c4be-4e18-9c37-9f6360b37ee1   tagged39             btbond2.39   VLAN            255.255.255.0      10.39.0.1          39       [IP Address on node0: 10.39.0.10]


The respective new tagged interface has been configured by odacli on the linux operating system :

[[email protected] ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*
-rw-r--r--. 1 root root 239 Feb  4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2
-rw-r--r--  1 root root 520 Feb  4 14:40 /etc/sysconfig/network-scripts/ifcfg-btbond2.38
-rw-r--r--  1 root root 520 Feb  4 14:45 /etc/sysconfig/network-scripts/ifcfg-btbond2.39

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.39
#ODA_VLAN_CONFIG ===
#ODA_VLAN_CONFIG Name=tagged39
#ODA_VLAN_CONFIG VlanId=39
#ODA_VLAN_CONFIG VlanInterface=btbond2
#ODA_VLAN_CONFIG Type=VlanType
#ODA_VLAN_CONFIG VlanSetupType=Other
#ODA_VLAN_CONFIG VlanIpAddr=10.39.0.10
#ODA_VLAN_CONFIG VlanNetmask=255.255.255.0
#ODA_VLAN_CONFIG VlanGateway=10.39.0.1
#ODA_VLAN_CONFIG NodeNum=0
#=== DO NOT EDIT ANYTHING ABOVE THIS LINE ===
DEVICE=btbond2.39
BOOTPROTO=none
ONBOOT=yes
VLAN=yes
NM_CONTROLLED=no
DEFROUTE=no
IPADDR=10.39.0.10
NETMASK=255.255.255.0
GATEWAY=10.39.0.1


And I can ping my new network as well :

C:Users>ping 10.39.0.10

Pinging 10.39.0.10 with 32 bytes of data:
Reply from 10.39.0.10: bytes=32 time=2ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63

Ping statistics for 10.39.0.10:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 2ms, Average = 0ms


I can now as well check my global btbond2 configuration on the linux side :

[[email protected] ~]# ip addr sh | grep -iE "(btbond2|btbond2.38|btbond2.39)"
4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
9: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000
57: [email protected]:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.38.0.10/24 brd 10.38.0.255 scope global btbond2.38
58: [email protected]:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.39.0.10/24 brd 10.39.0.255 scope global btbond2.39
[[email protected] ~]#


Creating virtual networks

Knowing I need to use the networks for the next DB Systems I have deleted the created physical networks and I’m going to create the 2 same tagged networks as virtual networks on the btbond2 interface :

[[email protected] ~]# odacli create-vnetwork -n tagged38 -if btbond2 -t bridged-vlan -ip 10.38.0.10 -nm 255.255.255.0 -vlan 38 -gw 10.38.0.1

Job details
----------------------------------------------------------------
                     ID:  eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1
            Description:  vNetwork tagged38 creation
                 Status:  Created
                Created:  February 4, 2022 3:19:02 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1

Job details
----------------------------------------------------------------
                     ID:  eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1
            Description:  vNetwork tagged38 creation
                 Status:  Success
                Created:  February 4, 2022 3:19:02 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network doesn‘t exist   February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success
Validate interface to use exists         February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success
Validate interfaces to create not exist  February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success
Create bridge                            February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success
Create VLAN                              February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:02 PM CET     Success
Bring up VLAN                            February 4, 2022 3:19:02 PM CET     February 4, 2022 3:19:07 PM CET     Success
Create metadata                          February 4, 2022 3:19:07 PM CET     February 4, 2022 3:19:07 PM CET     Success
Persist metadata                         February 4, 2022 3:19:07 PM CET     February 4, 2022 3:19:07 PM CET     Success

[[email protected] ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -ip 10.39.0.10 -nm 255.255.255.0 -vlan 39 -gw 10.39.0.1

Job details
----------------------------------------------------------------
                     ID:  15cf889f-9c6e-4e0c-a676-b2203e40cfd2
            Description:  vNetwork tagged39 creation
                 Status:  Created
                Created:  February 4, 2022 3:19:40 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i 15cf889f-9c6e-4e0c-a676-b2203e40cfd2

Job details
----------------------------------------------------------------
                     ID:  15cf889f-9c6e-4e0c-a676-b2203e40cfd2
            Description:  vNetwork tagged39 creation
                 Status:  Success
                Created:  February 4, 2022 3:19:40 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network doesn‘t exist   February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success
Validate interface to use exists         February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success
Validate interfaces to create not exist  February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success
Create bridge                            February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success
Create VLAN                              February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:40 PM CET     Success
Bring up VLAN                            February 4, 2022 3:19:40 PM CET     February 4, 2022 3:19:45 PM CET     Success
Create metadata                          February 4, 2022 3:19:45 PM CET     February 4, 2022 3:19:45 PM CET     Success
Persist metadata                         February 4, 2022 3:19:45 PM CET     February 4, 2022 3:19:45 PM CET     Success


My 2 new tagged networks exists as virtual network and not physical networks :

[[email protected] ~]# odacli list-networks

ID                                     Name                 NIC          Interface Type  Subnet Mask        Gateway            VLAN ID  Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d   Public-network       pubnet       BRIDGE          255.255.255.0      10.36.0.1                   [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b   ASM-network          privasm      BRIDGE          255.255.255.128                                [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25   Private-network      priv0        INTERNAL        255.255.255.240                                [IP Address on node0: 192.168.16.24]

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
tagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-04 15:19:45 CET  2022-02-04 15:19:45 CET
tagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET


Tagged network interfaces have been created by the dcs-agent on the linux operating side :

[[email protected] ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*
-rw-r--r--. 1 root root 239 Feb  4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2
-rw-r--r--  1 root root 145 Feb  4 15:19 /etc/sysconfig/network-scripts/ifcfg-btbond2.38
-rw-r--r--  1 root root 145 Feb  4 15:19 /etc/sysconfig/network-scripts/ifcfg-btbond2.39

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.38
#This file was created by ODA. Do not edit.
DEVICE=btbond2.38
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
VLAN=yes
ONPARENT=yes
BRIDGE=brtagged38

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.39
#This file was created by ODA. Do not edit.
DEVICE=btbond2.39
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
VLAN=yes
ONPARENT=yes
BRIDGE=brtagged39


On the linux side, I can see that the respective IP addresses have not been assigned to the physical interfaces :

[[email protected] ~]# ip addr sh | grep -iE "(btbond2|btbond2.38|btbond2.39)"
4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
9: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000
60: [email protected]:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
62: [email protected]:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000


But on new virtual interfaces linked to the tagged btbond2 interfaces :

[[email protected] ~]# ip addr sh | grep -iE "(tagged38|tagged39)"
59: brtagged38:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.38.0.10/24 brd 10.38.0.255 scope global brtagged38
60: [email protected]:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
61: brtagged39:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.39.0.10/24 brd 10.39.0.255 scope global brtagged39
62: [email protected]:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000


And I can ping them :

C:Users>ping 10.38.0.10

Pinging 10.38.0.10 with 32 bytes of data:
Reply from 10.38.0.10: bytes=32 time=2ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63

C:Users>ping 10.39.0.10

Pinging 10.39.0.10 with 32 bytes of data:
Reply from 10.39.0.10: bytes=32 time=2ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Reply from 10.39.0.10: bytes=32 time=1ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63

Ping statistics for 10.39.0.10:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 2ms, Average = 0ms


I can see the network bridge information from the operating system as well :

[[email protected] ~]# brctl show
bridge name	  bridge id		        STP enabled	       interfaces
brtagged38		8000.5254003bdc76	  no		             btbond2.38
							                                       vnet2
brtagged39		8000.52540060eb57	  no		             btbond2.39
							                                       vnet4
privasm		    8000.5a8241cdb508	  no		             priv0.100
							                                       vnet1
							                                       vnet3
pubnet		    8000.3cfdfe928018	  no	               btbond1
							                                       vnet0
virbr0		    8000.525400dc8c09	  yes		             virbr0-nic


Creating 2 DB Systems using respectively tagged38 and tagged39 virtual networks

My repository has already been updated with the KVM DB System Image :

[[email protected] ~]# odacli describe-dbsystem-image
DB System Image details
--------------------------------------------------------------------------------
Component Name        Supported Versions    Available Versions
--------------------  --------------------  --------------------

DBVM                  19.13.0.0.0           19.13.0.0.0

GI                    19.13.0.0.211019      19.13.0.0.211019
                      19.12.0.0.210720      not-available
                      19.11.0.0.210420      not-available
                      21.4.0.0.211019       not-available
                      21.3.0.0.210720       not-available

DB                    19.13.0.0.211019      19.13.0.0.211019
                      19.12.0.0.210720      not-available
                      19.11.0.0.210420      not-available
                      21.4.0.0.211019       not-available
                      21.3.0.0.210720       not-available


I have created the first DB System json file assigning a new IP address from the VLAN38 network using tagged38 virtual network. The IP connection will use bridging through the virtual VLAN38 IP address we created on the Bare Metal itself :

[[email protected] ~]# cat /opt/dbi/create_dbsystem_srvdb38.json
...
...
...
"network": {
    "domainName": "dbi-lab.ch",
    "ntpServers": ["216.239.35.0"],
    "dnsServers": [
        "8.8.8.8","8.8.4.4"
    ],
    "nodes": [
        {
            "name": "srvdb38",
            "ipAddress": "10.38.0.20",
            "netmask": "255.255.255.0",
            "gateway": "10.38.0.1",
            "number": 0
        }
    ],
"publicVNetwork": "tagged38"
},
"grid": {
    "language": "en"
}
}


Creation of the first DB System on network VLAN 38 :

[[email protected] ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb38.json
Enter password for system "srvdb38":
Retype password for system "srvdb38":
Enter administrator password for DB "DB38":
Retype administrator password for DB "DB38":

Job details
----------------------------------------------------------------
                     ID:  ed88ef81-5cb3-4214-ac5c-bc255b67577f
            Description:  DB System srvdb38 creation
                 Status:  Created
                Created:  February 4, 2022 4:12:33 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i ed88ef81-5cb3-4214-ac5c-bc255b67577f

Job details
----------------------------------------------------------------
                     ID:  ed88ef81-5cb3-4214-ac5c-bc255b67577f
            Description:  DB System srvdb38 creation
                 Status:  Success
                Created:  February 4, 2022 4:12:33 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create DB System metadata                February 4, 2022 4:12:33 PM CET     February 4, 2022 4:12:33 PM CET     Success
Persist new DB System                    February 4, 2022 4:12:33 PM CET     February 4, 2022 4:12:33 PM CET     Success
Validate DB System prerequisites         February 4, 2022 4:12:33 PM CET     February 4, 2022 4:12:37 PM CET     Success
Setup DB System environment              February 4, 2022 4:12:37 PM CET     February 4, 2022 4:12:39 PM CET     Success
Create DB System ASM volume              February 4, 2022 4:12:39 PM CET     February 4, 2022 4:12:45 PM CET     Success
Create DB System ACFS filesystem         February 4, 2022 4:12:45 PM CET     February 4, 2022 4:12:54 PM CET     Success
Create DB System VM ACFS snapshots       February 4, 2022 4:12:54 PM CET     February 4, 2022 4:13:27 PM CET     Success
Create temporary SSH key pair            February 4, 2022 4:13:27 PM CET     February 4, 2022 4:13:27 PM CET     Success
Create DB System cloud-init config       February 4, 2022 4:13:27 PM CET     February 4, 2022 4:13:27 PM CET     Success
Provision DB System VM(s)                February 4, 2022 4:13:27 PM CET     February 4, 2022 4:13:28 PM CET     Success
Attach disks to DB System                February 4, 2022 4:13:28 PM CET     February 4, 2022 4:13:29 PM CET     Success
Add DB System to Clusterware             February 4, 2022 4:13:29 PM CET     February 4, 2022 4:13:29 PM CET     Success
Start DB System                          February 4, 2022 4:13:29 PM CET     February 4, 2022 4:13:30 PM CET     Success
Wait DB System VM first boot             February 4, 2022 4:13:30 PM CET     February 4, 2022 4:17:06 PM CET     Success
Setup Mutual TLS (mTLS)                  February 4, 2022 4:17:06 PM CET     February 4, 2022 4:21:36 PM CET     Success
Export clones repository                 February 4, 2022 4:21:36 PM CET     February 4, 2022 4:21:37 PM CET     Success
Setup ASM client cluster config          February 4, 2022 4:21:37 PM CET     February 4, 2022 4:22:00 PM CET     Success
Install DB System                        February 4, 2022 4:22:00 PM CET     February 4, 2022 5:07:11 PM CET     Success
Cleanup temporary SSH key pair           February 4, 2022 5:07:11 PM CET     February 4, 2022 5:07:52 PM CET     Success
Set DB System as configured              February 4, 2022 5:07:52 PM CET     February 4, 2022 5:07:52 PM CET     Success


I have then created the second DB System json file assigning a new IP address from the VLAN39 network using tagged39 virtual network. That time the IP connection will use bridging through the virtual VLAN39 IP address we created on the Bare Metal itself :

[[email protected] ~]# cat /opt/dbi/create_dbsystem_srvdb39.json
...
...
...
},
"network": {
    "domainName": "dbi-lab.ch",
    "ntpServers": ["216.239.35.0"],
    "dnsServers": [
        "8.8.8.8","8.8.4.4"
    ],
    "nodes": [
        {
            "name": "srvdb39",
            "ipAddress": "10.39.0.20",
            "netmask": "255.255.255.0",
            "gateway": "10.39.0.1",
            "number": 0
        }
    ],
"publicVNetwork": "tagged39"
},
"grid": {
    "language": "en"
}
}


Creation on the second DB System on network VLAN 39 :

[[email protected] ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb39.json
Enter password for system "srvdb39":
Retype password for system "srvdb39":
Enter administrator password for DB "DB39":
Retype administrator password for DB "DB39":

Job details
----------------------------------------------------------------
                     ID:  38303453-8524-4c65-b1d2-3717dfc79a1f
            Description:  DB System srvdb39 creation
                 Status:  Created
                Created:  February 4, 2022 5:14:01 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i 38303453-8524-4c65-b1d2-3717dfc79a1f

Job details
----------------------------------------------------------------
                     ID:  38303453-8524-4c65-b1d2-3717dfc79a1f
            Description:  DB System srvdb39 creation
                 Status:  Success
                Created:  February 4, 2022 5:14:01 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create DB System metadata                February 4, 2022 5:14:01 PM CET     February 4, 2022 5:14:01 PM CET     Success
Persist new DB System                    February 4, 2022 5:14:01 PM CET     February 4, 2022 5:14:01 PM CET     Success
Validate DB System prerequisites         February 4, 2022 5:14:01 PM CET     February 4, 2022 5:14:05 PM CET     Success
Setup DB System environment              February 4, 2022 5:14:05 PM CET     February 4, 2022 5:14:07 PM CET     Success
Create DB System ASM volume              February 4, 2022 5:14:07 PM CET     February 4, 2022 5:14:13 PM CET     Success
Create DB System ACFS filesystem         February 4, 2022 5:14:13 PM CET     February 4, 2022 5:14:22 PM CET     Success
Create DB System VM ACFS snapshots       February 4, 2022 5:14:22 PM CET     February 4, 2022 5:14:52 PM CET     Success
Create temporary SSH key pair            February 4, 2022 5:14:52 PM CET     February 4, 2022 5:14:52 PM CET     Success
Create DB System cloud-init config       February 4, 2022 5:14:52 PM CET     February 4, 2022 5:14:53 PM CET     Success
Provision DB System VM(s)                February 4, 2022 5:14:53 PM CET     February 4, 2022 5:14:54 PM CET     Success
Attach disks to DB System                February 4, 2022 5:14:54 PM CET     February 4, 2022 5:14:54 PM CET     Success
Add DB System to Clusterware             February 4, 2022 5:14:54 PM CET     February 4, 2022 5:14:54 PM CET     Success
Start DB System                          February 4, 2022 5:14:54 PM CET     February 4, 2022 5:14:56 PM CET     Success
Wait DB System VM first boot             February 4, 2022 5:14:56 PM CET     February 4, 2022 5:18:32 PM CET     Success
Setup Mutual TLS (mTLS)                  February 4, 2022 5:18:32 PM CET     February 4, 2022 5:23:07 PM CET     Success
Export clones repository                 February 4, 2022 5:23:07 PM CET     February 4, 2022 5:23:07 PM CET     Success
Setup ASM client cluster config          February 4, 2022 5:23:07 PM CET     February 4, 2022 5:23:30 PM CET     Success
Install DB System                        February 4, 2022 5:23:30 PM CET     February 4, 2022 6:09:43 PM CET     Success
Cleanup temporary SSH key pair           February 4, 2022 6:09:43 PM CET     February 4, 2022 6:10:23 PM CET     Success
Set DB System as configured              February 4, 2022 6:10:23 PM CET     February 4, 2022 6:10:23 PM CET     Success


Virtual network configuration checks

So I have got my both virtual networks configured on respective IP address (10.38.0.10 for VLAN38 network and 10.39.0.10 for VLAN39 network) :

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
tagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-04 15:19:45 CET  2022-02-04 15:19:45 CET
tagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET

[[email protected] ~]# odacli describe-vnetwork -n tagged38
VNetwork details
--------------------------------------------------------------------------------
                       ID:  0d8099ba-ff42-4d73-82b4-f7497fb68ca5
                     Name:  tagged38
                  Created:  2022-02-04 15:19:07 CET
                  Updated:  2022-02-04 15:19:07 CET
                     Type:  BridgedVlan
           Interface name:  btbond2
              Bridge name:  brtagged38
                  VLAN ID:  38
                       IP:  10.38.0.10
                  Netmask:  255.255.255.0
                  Gateway:  10.38.0.1
 Attached in VMs (config):  xa287c764d
   Attached in VMs (live):  xa287c764d

[[email protected] ~]# odacli describe-vnetwork -n tagged39
VNetwork details
--------------------------------------------------------------------------------
                       ID:  00df04d4-9a01-4547-9a3c-5a27dab10494
                     Name:  tagged39
                  Created:  2022-02-04 15:19:45 CET
                  Updated:  2022-02-04 15:19:45 CET
                     Type:  BridgedVlan
           Interface name:  btbond2
              Bridge name:  brtagged39
                  VLAN ID:  39
                       IP:  10.39.0.10
                  Netmask:  255.255.255.0
                  Gateway:  10.39.0.1
 Attached in VMs (config):  x793d6c5ce
   Attached in VMs (live):  x793d6c5ce


DB System information checks

I have my 2 DB Systems.
srvdb38 configured on VLAN38 network with 10.38.0.20 IP Address.
srvdb39 configured on VLAN39 network with 10.39.0.20 IP Address.

[[email protected] ~]# odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb39               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 17:14:01 CET  2022-02-04 18:10:23 CET
srvdb38               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 16:12:33 CET  2022-02-04 17:07:52 CET

[[email protected] ~]# odacli describe-dbsystem -n srvdb38
DB System details
--------------------------------------------------------------------------------
                       ID:  859466fe-a4a5-40ae-aa00-8e02cf74dedc
                     Name:  srvdb38
                    Image:  19.13.0.0.0
                    Shape:  odb2
             Cluster name:  dbsa287c764d
             Grid version:  19.13.0.0.211019
                   Memory:  16.00 GB
             NUMA enabled:  YES
                   Status:  CONFIGURED
                  Created:  2022-02-04 16:12:33 CET
                  Updated:  2022-02-04 17:07:52 CET

 CPU Pool
--------------------------
                     Name:  cpupool4dbsystems
          Number of cores:  4

                     Host:  dbi-oda-x8
        Effective CPU set:  5-6,21-22,37-38,53-54
              Online CPUs:  5, 6, 21, 22, 37, 38, 53, 54
             Offline CPUs:  NONE

 VM Storage
--------------------------
               Disk group:  DATA
              Volume name:  SA287C764D
            Volume device:  /dev/asm/sa287c764d-390
                     Size:  200.00 GB
              Mount Point:  /u05/app/sharedrepo/srvdb38

 VMs
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  xa287c764d
             VM Host Name:  srvdb38.dbi-lab.ch
            VM image path:  /u05/app/sharedrepo/srvdb38/.ACFS/snaps/vm_xa287c764d/xa287c764d
             Target State:  ONLINE
            Current State:  ONLINE

 VNetworks
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  xa287c764d
                   Public:  10.38.0.20      / 255.255.255.0   / ens3 / BRIDGE(brtagged38)
                      ASM:  192.168.17.4    / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)

 Extra VNetworks
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  xa287c764d
                 tagged38:  10.38.0.20      / 255.255.255.0   / PUBLIC

 Databases
--------------------------
                     Name:  DB38
              Resource ID:  5ff22777-f24b-446a-8c58-3fa6dd2ad383
              Unique name:  DB38_SITE1
              Database ID:  715007309
              Domain name:  dbi-lab.ch
               DB Home ID:  8b66a7d6-9e5e-43c0-a69c-00b80d2dba81
                    Shape:  odb2
                  Version:  19.13.0.0.211019
                  Edition:  EE
                     Type:  SI
                     Role:  PRIMARY
                    Class:  OLTP
                  Storage:  ASM
               Redundancy:
         Target node name:
            Character set:  AL32UTF8
        NLS character set:
                 Language:  ENGLISH
                Territory:  AMERICA
          Console enabled:  false
             SEHA enabled:  false
      Associated networks:  Public-network
         Backup config ID:
       Level 0 Backup Day:  sunday
       Autobackup enabled:  true
              TDE enabled:  false
                 CDB type:  false
                 PDB name:
           PDB admin user:

[[email protected] ~]# odacli describe-dbsystem -n srvdb39
DB System details
--------------------------------------------------------------------------------
                       ID:  5238f074-6126-4051-8b9e-e392b55a2328
                     Name:  srvdb39
                    Image:  19.13.0.0.0
                    Shape:  odb2
             Cluster name:  dbs793d6c5ce
             Grid version:  19.13.0.0.211019
                   Memory:  16.00 GB
             NUMA enabled:  YES
                   Status:  CONFIGURED
                  Created:  2022-02-04 17:14:01 CET
                  Updated:  2022-02-04 18:10:23 CET

 CPU Pool
--------------------------
                     Name:  cpupool4dbsystems
          Number of cores:  4

                     Host:  dbi-oda-x8
        Effective CPU set:  5-6,21-22,37-38,53-54
              Online CPUs:  5, 6, 21, 22, 37, 38, 53, 54
             Offline CPUs:  NONE

 VM Storage
--------------------------
               Disk group:  DATA
              Volume name:  S793D6C5CE
            Volume device:  /dev/asm/s793d6c5ce-390
                     Size:  200.00 GB
              Mount Point:  /u05/app/sharedrepo/srvdb39

 VMs
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  x793d6c5ce
             VM Host Name:  srvdb39.dbi-lab.ch
            VM image path:  /u05/app/sharedrepo/srvdb39/.ACFS/snaps/vm_x793d6c5ce/x793d6c5ce
             Target State:  ONLINE
            Current State:  ONLINE

 VNetworks
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  x793d6c5ce
                   Public:  10.39.0.20      / 255.255.255.0   / ens3 / BRIDGE(brtagged39)
                      ASM:  192.168.17.5    / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)

 Extra VNetworks
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  x793d6c5ce
                 tagged39:  10.39.0.20      / 255.255.255.0   / PUBLIC

 Databases
--------------------------
                     Name:  DB39
              Resource ID:  e796e7e1-c8c8-4a63-809c-57976ce2163d
              Unique name:  DB39_SITE1
              Database ID:  2043690133
              Domain name:  dbi-lab.ch
               DB Home ID:  67f6df00-83d8-477b-8884-201664a3701b
                    Shape:  odb2
                  Version:  19.13.0.0.211019
                  Edition:  EE
                     Type:  SI
                     Role:  PRIMARY
                    Class:  OLTP
                  Storage:  ASM
               Redundancy:
         Target node name:
            Character set:  AL32UTF8
        NLS character set:
                 Language:  ENGLISH
                Territory:  AMERICA
          Console enabled:  false
             SEHA enabled:  false
      Associated networks:  Public-network
         Backup config ID:
       Level 0 Backup Day:  sunday
       Autobackup enabled:  true
              TDE enabled:  false
                 CDB type:  false
                 PDB name:
           PDB admin user:

[[email protected] ~]#


Direct connection on the DB Systems

Finally I can access directly the DB Systems through SSH from my laptop. Here as example, connection to the srvdb38 DB System.

[email protected] ~ % ssh [email protected]
[email protected]'s password:
Last login: Thu Feb 17 16:07:31 2022

[[email protected] ~]# ip addr sh ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:6e:3f:9f brd ff:ff:ff:ff:ff:ff
    inet 10.38.0.20/24 brd 10.38.0.255 scope global noprefixroute ens3
       valid_lft forever preferred_lft forever
       
[[email protected] ~]# ps -ef | grep [p]mon
oracle   21506     1  0 14:05 ?        00:00:00 ora_pmon_DB38
[[email protected] ~]#


And directly from my laptop I can connect and use the Oracle database DB38 :








Creating a virtual network without assigning an IP address to the bridge interface

Of course it is possible to create a virtual bridge interface without assigning any IP address.

I deleted tagged39 virtual network in order to create it again without IP address :

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
tagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET
tagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-04 22:13:18 CET  2022-02-17 16:38:02 CET

[[email protected] ~]# odacli delete-vnetwork -n tagged39

Job details
----------------------------------------------------------------
                     ID:  2cae44c4-39e7-4102-8453-597f707ed10b
            Description:  vNetwork tagged39 deletion
                 Status:  Created
                Created:  February 17, 2022 4:46:45 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i 2cae44c4-39e7-4102-8453-597f707ed10b

Job details
----------------------------------------------------------------
                     ID:  2cae44c4-39e7-4102-8453-597f707ed10b
            Description:  vNetwork tagged39 deletion
                 Status:  Success
                Created:  February 17, 2022 4:46:45 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network exists          February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:45 PM CET    Success
Validate VM does not have the VNetwork attached February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:45 PM CET    Success
Delete bridge                            February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:45 PM CET    Success
Delete VLAN                              February 17, 2022 4:46:45 PM CET    February 17, 2022 4:46:46 PM CET    Success
Delete metadata                          February 17, 2022 4:46:46 PM CET    February 17, 2022 4:46:46 PM CET    Success

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
tagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET

[[email protected] ~]# ip addr sh | grep btbond2
4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
10: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000
11: [email protected]:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
[[email protected] ~]#


Of course I could have updated the vnetwork interface using odacli modify-vnetwork and assigning 0.0.0.0 as IP address.

I created again the tagged39 virtual network without assigning an IP address to the bridge interface.

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
tagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET

[[email protected] ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -vlan 39

Job details
----------------------------------------------------------------
                     ID:  a70a1313-f2cd-48ab-ac5f-c17b2f2fae08
            Description:  vNetwork tagged39 creation
                 Status:  Created
                Created:  February 17, 2022 4:49:44 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i a70a1313-f2cd-48ab-ac5f-c17b2f2fae08

Job details
----------------------------------------------------------------
                     ID:  a70a1313-f2cd-48ab-ac5f-c17b2f2fae08
            Description:  vNetwork tagged39 creation
                 Status:  Success
                Created:  February 17, 2022 4:49:44 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network doesn't exist   February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success
Validate interface to use exists         February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success
Validate interfaces to create not exist  February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success
Create bridge                            February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success
Create VLAN                              February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:44 PM CET    Success
Bring up VLAN                            February 17, 2022 4:49:44 PM CET    February 17, 2022 4:49:45 PM CET    Success
Create metadata                          February 17, 2022 4:49:45 PM CET    February 17, 2022 4:49:45 PM CET    Success
Persist metadata                         February 17, 2022 4:49:45 PM CET    February 17, 2022 4:49:45 PM CET    Success

[[email protected] ~]# odacli list-vnetworks
Name                  Type             Interface        Bridge                Uniform   Created                  Updated
--------------------  ---------------  ---------------  --------------------  --------  -----------------------  -----------------------
tagged38              BridgedVlan      btbond2          brtagged38            NO        2022-02-04 15:19:07 CET  2022-02-04 15:19:07 CET
pubnet                Bridged          btbond1          pubnet                NO        2022-01-28 08:54:55 CET  2022-01-28 08:54:55 CET
tagged39              BridgedVlan      btbond2          brtagged39            NO        2022-02-17 16:49:45 CET  2022-02-17 16:49:45 CET

[[email protected] ~]# odacli describe-vnetwork -n tagged39
VNetwork details
--------------------------------------------------------------------------------
                       ID:  6e40ca21-1d4a-4bcc-8436-f29ffb2eb8bc
                     Name:  tagged39
                  Created:  2022-02-17 16:49:45 CET
                  Updated:  2022-02-17 16:49:45 CET
                     Type:  BridgedVlan
           Interface name:  btbond2
              Bridge name:  brtagged39
                  VLAN ID:  39
                       IP:
                  Netmask:
                  Gateway:
 Attached in VMs (config):  NONE
   Attached in VMs (live):  NONE

[[email protected] ~]# ip addr sh | grep btbond2
4: p7p3:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4:  mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
10: btbond2:  mtu 1500 qdisc noqueue state UP group default qlen 1000
11: [email protected]:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
31: [email protected]:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000

[[email protected] ~]# ip addr sh | grep brtagged
11: [email protected]:  mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
12: brtagged38:  mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.38.0.10/24 brd 10.38.0.255 scope global brtagged38
25: vnet6:  mtu 1500 qdisc pfifo_fast master brtagged38 state UNKNOWN group default qlen 1000
30: brtagged39:  mtu 1500 qdisc noqueue state UP group default qlen 1000
31: [email protected]:  mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000
[[email protected] ~]#


And let’s deploy the srvdb39 DB Systems again after having removed it with odacli delete-dbsystem command :

[[email protected] ~]# odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb38               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 16:12:33 CET  2022-02-04 17:07:52 CET

[[email protected] ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb39.json
Enter password for system "srvdb39":
Retype password for system "srvdb39":
Enter administrator password for DB "DB39":
Retype administrator password for DB "DB39":

Job details
----------------------------------------------------------------
                     ID:  934c752b-7eb6-4b5b-8856-7a90102a1e65
            Description:  DB System srvdb39 creation
                 Status:  Created
                Created:  February 17, 2022 5:06:37 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] ~]# odacli describe-job -i 934c752b-7eb6-4b5b-8856-7a90102a1e65

Job details
----------------------------------------------------------------
                     ID:  934c752b-7eb6-4b5b-8856-7a90102a1e65
            Description:  DB System srvdb39 creation
                 Status:  Success
                Created:  February 17, 2022 5:06:37 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create DB System metadata                February 17, 2022 5:06:37 PM CET    February 17, 2022 5:06:37 PM CET    Success
Persist new DB System                    February 17, 2022 5:06:37 PM CET    February 17, 2022 5:06:37 PM CET    Success
Validate DB System prerequisites         February 17, 2022 5:06:37 PM CET    February 17, 2022 5:06:41 PM CET    Success
Setup DB System environment              February 17, 2022 5:06:41 PM CET    February 17, 2022 5:06:43 PM CET    Success
Create DB System ASM volume              February 17, 2022 5:06:43 PM CET    February 17, 2022 5:06:49 PM CET    Success
Create DB System ACFS filesystem         February 17, 2022 5:06:49 PM CET    February 17, 2022 5:06:58 PM CET    Success
Create DB System VM ACFS snapshots       February 17, 2022 5:06:58 PM CET    February 17, 2022 5:07:31 PM CET    Success
Create temporary SSH key pair            February 17, 2022 5:07:31 PM CET    February 17, 2022 5:07:32 PM CET    Success
Create DB System cloud-init config       February 17, 2022 5:07:32 PM CET    February 17, 2022 5:07:32 PM CET    Success
Provision DB System VM(s)                February 17, 2022 5:07:32 PM CET    February 17, 2022 5:07:33 PM CET    Success
Attach disks to DB System                February 17, 2022 5:07:33 PM CET    February 17, 2022 5:07:34 PM CET    Success
Add DB System to Clusterware             February 17, 2022 5:07:34 PM CET    February 17, 2022 5:07:34 PM CET    Success
Start DB System                          February 17, 2022 5:07:34 PM CET    February 17, 2022 5:07:36 PM CET    Success
Wait DB System VM first boot             February 17, 2022 5:07:36 PM CET    February 17, 2022 5:08:48 PM CET    Success
Setup Mutual TLS (mTLS)                  February 17, 2022 5:08:48 PM CET    February 17, 2022 5:09:08 PM CET    Success
Export clones repository                 February 17, 2022 5:09:08 PM CET    February 17, 2022 5:09:08 PM CET    Success
Setup ASM client cluster config          February 17, 2022 5:09:08 PM CET    February 17, 2022 5:09:11 PM CET    Success
Install DB System                        February 17, 2022 5:09:11 PM CET    February 17, 2022 5:36:57 PM CET    Success
Cleanup temporary SSH key pair           February 17, 2022 5:36:57 PM CET    February 17, 2022 5:36:57 PM CET    Success
Set DB System as configured              February 17, 2022 5:36:57 PM CET    February 17, 2022 5:36:57 PM CET    Success

[[email protected] ~]# odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb38               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-04 16:12:33 CET  2022-02-04 17:07:52 CET
srvdb39               odb2        4      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-17 17:06:37 CET  2022-02-17 17:36:57 CET

[[email protected] ~]# odacli describe-dbsystem -n srvdb39
DB System details
--------------------------------------------------------------------------------
                       ID:  ed13bacb-be28-4e31-a5d7-39fd713d72a0
                     Name:  srvdb39
                    Image:  19.13.0.0.0
                    Shape:  odb2
             Cluster name:  dbs793d6c5ce
             Grid version:  19.13.0.0.211019
                   Memory:  16.00 GB
             NUMA enabled:  YES
                   Status:  CONFIGURED
                  Created:  2022-02-17 17:06:37 CET
                  Updated:  2022-02-17 17:36:57 CET

 CPU Pool
--------------------------
                     Name:  cpupool4dbsystems
          Number of cores:  4

                     Host:  dbi-oda-x8
        Effective CPU set:  5-6,21-22,37-38,53-54
              Online CPUs:  5, 6, 21, 22, 37, 38, 53, 54
             Offline CPUs:  NONE

 VM Storage
--------------------------
               Disk group:  DATA
              Volume name:  S793D6C5CE
            Volume device:  /dev/asm/s793d6c5ce-390
                     Size:  200.00 GB
              Mount Point:  /u05/app/sharedrepo/srvdb39

 VMs
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  x793d6c5ce
             VM Host Name:  srvdb39.dbi-lab.ch
            VM image path:  /u05/app/sharedrepo/srvdb39/.ACFS/snaps/vm_x793d6c5ce/x793d6c5ce
             Target State:  ONLINE
            Current State:  ONLINE

 VNetworks
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  x793d6c5ce
                   Public:  10.39.0.20      / 255.255.255.0   / ens3 / BRIDGE(brtagged39)
                      ASM:  192.168.17.5    / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)

 Extra VNetworks
--------------------------
                     Host:  dbi-oda-x8
                  VM Name:  x793d6c5ce
                 tagged39:  10.39.0.20      / 255.255.255.0   / PUBLIC

 Databases
--------------------------
                     Name:  DB39
              Resource ID:  6ecee323-cd18-4971-acb4-b476d4d76d2c
              Unique name:  DB39_SITE1
              Database ID:  2044811647
              Domain name:  dbi-lab.ch
               DB Home ID:  3a6f8134-407a-4272-8512-7c52bdca68ea
                    Shape:  odb2
                  Version:  19.13.0.0.211019
                  Edition:  EE
                     Type:  SI
                     Role:  PRIMARY
                    Class:  OLTP
                  Storage:  ASM
               Redundancy:
         Target node name:
            Character set:  AL32UTF8
        NLS character set:
                 Language:  ENGLISH
                Territory:  AMERICA
          Console enabled:  false
             SEHA enabled:  false
      Associated networks:  Public-network
         Backup config ID:
       Level 0 Backup Day:  sunday
       Autobackup enabled:  true
              TDE enabled:  false
                 CDB type:  false
                 PDB name:
           PDB admin user:


This DB System is also now accessible through his IP address 10.39.0.20 and is using the bridge interface without any IP address. A bridge interface does not need any IP address.

I can ping the new DB System from my laptop :

[email protected] ~ % ping 10.39.0.20
PING 10.39.0.20 (10.39.0.20): 56 data bytes
64 bytes from 10.39.0.20: icmp_seq=0 ttl=62 time=56.761 ms
64 bytes from 10.39.0.20: icmp_seq=1 ttl=62 time=56.734 ms
64 bytes from 10.39.0.20: icmp_seq=2 ttl=62 time=64.759 ms
^C
--- 10.39.0.20 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 56.734/59.418/64.759/3.777 ms


I can ssh the new DB System from my laptop :

[email protected] ~ % ssh [email protected]
[email protected]0's password:
Last login: Thu Feb 17 18:10:10 2022 from gateway

[[email protected] ~]# ip addr sh ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:86:87:88 brd ff:ff:ff:ff:ff:ff
    inet 10.39.0.20/24 brd 10.39.0.255 scope global noprefixroute ens3
       valid_lft forever preferred_lft forever
       
[[email protected] ~]# ps -ef | grep [p]mon
oracle   93401     1  0 17:35 ?        00:00:00 ora_pmon_DB39
[[email protected] ~]#


And I can join the DB39 database from my laptop directly :





















Conclusion

KVM DB System main purpose is hard partitioning for oracle licensing. All ODA’s cpu cores can remain active. DB System with appropriate pinned CPU core related to the number of licenses will be created for the oracle database. The others ODA’s CPU core can be used for further KVM or application needs.
As we could see KVM DB System will then have an additional pros. With DB System we can separate Database traffic through separated VLANs.

On the other hand KVM DB System will have some other cons :

  • More complex installation.
  • More resources will be consumed : each DB Systems will run its own oracle restart (grid) infrastructure.
  • Each DB Systems will have to be patched separately as well during ODA patching, making patching process more heavy.

Also please note that current ODA release 19.13 supports only creation of database in version 19.11, 19.12, 19.13, 21.3 and 21.4 for the DB Systems.