Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how to create additional network on the ODA and how to create DB System on separate VLAN interface. The principle would be of course the same if we would like to create KVM Virtual Machines (compute instance) on separate VLAN.
Configuration description
For my test I have got an ODA X8-2M with a quad-port 10GBase-T network interface. I’m running ODA version 19.13.
On my network card, the 2 first ports p7p1 and p7p2 will be assigned to btbond1 and the 2 next ports p7p3 and p7p4 will be assigned to my second bonding interface btbond2. ODA are configured by default with active-backup mode without LACP for all bonding interfaces. This is automatically designed by the ODA and can not be changed. More over we need to keep in mind that on an appliance we will never manually change the Linux network scripts. All network configuration changes needs to be done with odacli.
btbond1 is used for my main network and we will use btbond2 to add additionnal networks.
The 2 additional networks are :
10.38.0.1/24 VLAN id 38
10.39.0.1/24 VLAN id 39
Checking network interface
With ethtool I can check and see that the 2 p7p3 and p7p4 ports are twisted pair and connected to the network :
[root@dbi-oda-x8 ~]# ethtool p7p3 | grep -iE "(bportb|detected)" Port: Twisted Pair Link detected: yes [root@dbi-oda-x8 ~]# ethtool p7p4 | grep -iE "(bportb|detected)" Port: Twisted Pair Link detected: yes
I can see that both those interfaces are assigned to btbond2 interface :
[root@dbi-oda-x8 ~]# grep MASTER /etc/sysconfig/network-scripts/ifcfg-p7p3 MASTER=btbond2 [root@dbi-oda-x8 ~]# grep MASTER /etc/sysconfig/network-scripts/ifcfg-p7p4 MASTER=btbond2
Checking existing and default ODA configuration
After reimaging an ODA we would have the below default configuration.
root@dbi-oda-x8 ~]# odacli list-networks ID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks -------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- ----------------------- bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241] 95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2] e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24] [root@dbi-oda-x8 ~]# odacli list-vnetworks Name Type Interface Bridge Uniform Created Updated -------------------- --------------- --------------- -------------------- -------- ----------------------- ----------------------- pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET [root@dbi-oda-x8 ~]#
To use a network interface with the VMs, either a DB System or a Compute instance, the network interface needs to be created in the vnetworks.
VLAN Tagged versus Untagged
If a port on a switch is configured as tagged (trunk port in cisco world), the connected equipment (here our ODA) is VLAN aware, and would need to be tagged as well. The port is enabled for VLAN tagging. Purpose is to pass traffic for multiple VLAN’s. The ethernet frame is enclosing an additional VLAN header. The connected equipment will add the VLAN information in the ethernet frame.
If a port is configured as untagged (access port in cisco world), the connected equipment (here our ODA) does not know anything about in which VLAN it is and does not care about it. The switch will manage it on its own. The port does not tag and only accepts a single VLAN. The ethernet frame coming to the connected equipment will not have any VLAN header.
This is explained in the 802.1Q standard.
Most of the time trunk ports will link switches and access ports will link to end devices, albeit we would like to use several VLAN network on the same ODA network interface.
Creating an additional untagged network
Let’s create an additional untagged physical network on the Bare Metal itself. We will create it on btbond2 interface. Untagged means that the ODA will not add VLAN information in the Ethernet frame.
The option -t bond would be the default one and would here not be required.
[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m untagged1 -s 255.255.255.0 -g 10.38.0.1 -t bond
{
"jobId" : "9a4476dd-b955-433b-9463-377f66ab737a",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "February 04, 2022 13:48:12 PM CET",
"resourceList" : [ ],
"description" : "Rac Network service creation with name untagged1",
"updatedTime" : "February 04, 2022 13:48:12 PM CET"
}
[root@dbi-oda-x8 ~]# odacli describe-job -i 9a4476dd-b955-433b-9463-377f66ab737a
Job details
----------------------------------------------------------------
ID: 9a4476dd-b955-433b-9463-377f66ab737a
Description: Rac Network service creation with name untagged1
Status: Success
Created: February 4, 2022 1:48:12 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting network February 4, 2022 1:48:12 PM CET February 4, 2022 1:48:18 PM CET Success
Setting up Network February 4, 2022 1:48:12 PM CET February 4, 2022 1:48:12 PM CET Success
restart network interface btbond2 February 4, 2022 1:48:12 PM CET February 4, 2022 1:48:18 PM CET Success
[root@dbi-oda-x8 ~]# odacli list-networks
ID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks
-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------
bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]
95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]
e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24]
6171aad7-e247-4b08-a56e-83bdedd74af1 untagged1 btbond2 BOND 255.255.255.0 10.38.0.1 [IP Address on node0: 10.38.0.10]
[root@dbi-oda-x8 ~]# odacli list-vnetworks
Name Type Interface Bridge Uniform Created Updated
-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------
pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
I can see that our new network exists in the network list but not in the virtual network of course.
I can also check and see that our btbond2 interface has been configured by odacli as an untagged bonding interface with the appropriate information.
[root@dbi-oda-x8 ~]# ip addr sh btbond2
9: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 3c:fd:fe:92:80:1a brd ff:ff:ff:ff:ff:ff
inet 10.38.0.10/24 brd 10.38.0.255 scope global btbond2
valid_lft forever preferred_lft forever
[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*
-rw-r--r--. 1 root root 262 Feb 4 13:48 /etc/sysconfig/network-scripts/ifcfg-btbond2
[root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2
#This file was created by ODA. Do not edit.
NETMASK=255.255.255.0
GATEWAY=10.38.0.1
BOOTPROTO=none
PEERDNS=no
DEVICE=btbond2
ONBOOT=yes
NM_CONTROLLED=no
IPADDR=10.38.0.10
BONDING_OPTS="mode=active-backup miimon=100 primary=p7p3"
IPV6INIT=no
USERCTL=no
TYPE=BOND
Of course it is not possible to create another untagged network on btbond2 interface knowing there is already one existing :
[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bond DCS-10001:Internal error encountered: nicnamebtbond2 already exists in the networks list .. .
Creating an additional tagged network would also not be possible. Note the option -t and -v to create tagged networks on the ODA :
[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged2 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39 DCS-10001:Internal error encountered: Creating vlan in the interface btbond2 is not allowed. Physical network untagged1 already exists in interface btbond2.
Let’s delete our additional untagged network
[root@dbi-oda-x8 ~]# odacli delete-network -m untagged1
{
"jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
"status" : "Running",
"message" : null,
"reports" : [ {
"taskId" : "TaskSequential_10041",
"taskName" : "deleting network",
"taskResult" : "",
"startTime" : "February 04, 2022 14:08:32 PM CET",
"endTime" : "February 04, 2022 14:08:32 PM CET",
"status" : "Running",
"taskDescription" : null,
"parentTaskId" : "TaskSequential_10039",
"jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
"tags" : [ ],
"reportLevel" : "Info",
"updatedTime" : "February 04, 2022 14:08:32 PM CET"
}, {
"taskId" : "TaskZJsonRpcExt_10045",
"taskName" : "Setting up Network",
"taskResult" : "Network setup success",
"startTime" : "February 04, 2022 14:08:32 PM CET",
"endTime" : "February 04, 2022 14:08:32 PM CET",
"status" : "Success",
"taskDescription" : null,
"parentTaskId" : "TaskSequential_10041",
"jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
"tags" : [ ],
"reportLevel" : "Info",
"updatedTime" : "February 04, 2022 14:08:32 PM CET"
}, {
"taskId" : "TaskZJsonRpcExt_10048",
"taskName" : "restart network interface btbond2",
"taskResult" : "",
"startTime" : "February 04, 2022 14:08:32 PM CET",
"endTime" : "February 04, 2022 14:08:32 PM CET",
"status" : "Running",
"taskDescription" : null,
"parentTaskId" : "TaskSequential_10047",
"jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
"tags" : [ ],
"reportLevel" : "Info",
"updatedTime" : "February 04, 2022 14:08:32 PM CET"
} ],
"createTimestamp" : "February 04, 2022 14:08:32 PM CET",
"resourceList" : [ {
"resourceId" : "6171aad7-e247-4b08-a56e-83bdedd74af1",
"resourceType" : null,
"resourceNewType" : "Network",
"jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57",
"updatedTime" : null
} ],
"description" : "Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1",
"updatedTime" : "February 04, 2022 14:08:32 PM CET"
}
[root@dbi-oda-x8 ~]# odacli describe-job -i 3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57
Job details
----------------------------------------------------------------
ID: 3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57
Description: Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1
Status: Success
Created: February 4, 2022 2:08:32 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
deleting network February 4, 2022 2:08:32 PM CET February 4, 2022 2:08:34 PM CET Success
Setting up Network February 4, 2022 2:08:32 PM CET February 4, 2022 2:08:32 PM CET Success
restart network interface btbond2 February 4, 2022 2:08:32 PM CET February 4, 2022 2:08:34 PM CET Success
Network has been deleted :
[root@dbi-oda-x8 ~]# odacli list-networks ID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks -------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- ----------------------- bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241] 95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2] e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24]
Creating additional tagged network
Let’s create the first tagged network on btbond2 interface :
[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m tagged38 -s 255.255.255.0 -g 10.38.0.1 -t VLAN -v 38
{
"jobId" : "b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "February 04, 2022 14:40:28 PM CET",
"resourceList" : [ ],
"description" : "Rac Network service creation with name tagged38",
"updatedTime" : "February 04, 2022 14:40:28 PM CET"
}
[root@dbi-oda-x8 ~]# odacli describe-job -i b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc
Job details
----------------------------------------------------------------
ID: b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc
Description: Rac Network service creation with name tagged38
Status: Success
Created: February 4, 2022 2:40:28 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting network February 4, 2022 2:40:28 PM CET February 4, 2022 2:40:33 PM CET Success
Setting up Vlan February 4, 2022 2:40:28 PM CET February 4, 2022 2:40:33 PM CET Success
The tagged network has been created :
[root@dbi-oda-x8 ~]# odacli list-networks ID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks -------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- ----------------------- bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241] 95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2] e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24] fbe05bfa-636e-4f9c-a348-c59ba23e2296 tagged38 btbond2.38 VLAN 255.255.255.0 10.38.0.1 38 [IP Address on node0: 10.38.0.10]
And we can see that respective tagged network interface has now been created on the Linux operating system side (note the <btbond>.<vlan_id>) :
[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2* -rw-r--r--. 1 root root 239 Feb 4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2 -rw-r--r-- 1 root root 520 Feb 4 14:40 /etc/sysconfig/network-scripts/ifcfg-btbond2.38 [root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.38 #ODA_VLAN_CONFIG === #ODA_VLAN_CONFIG Name=tagged38 #ODA_VLAN_CONFIG VlanId=38 #ODA_VLAN_CONFIG VlanInterface=btbond2 #ODA_VLAN_CONFIG Type=VlanType #ODA_VLAN_CONFIG VlanSetupType=Other #ODA_VLAN_CONFIG VlanIpAddr=10.38.0.10 #ODA_VLAN_CONFIG VlanNetmask=255.255.255.0 #ODA_VLAN_CONFIG VlanGateway=10.38.0.1 #ODA_VLAN_CONFIG NodeNum=0 #=== DO NOT EDIT ANYTHING ABOVE THIS LINE === DEVICE=btbond2.38 BOOTPROTO=none ONBOOT=yes VLAN=yes NM_CONTROLLED=no DEFROUTE=no IPADDR=10.38.0.10 NETMASK=255.255.255.0 GATEWAY=10.38.0.1 [root@dbi-oda-x8 ~]#
And I can reach the ODA on the new network :
C:Users>ping 10.38.0.10
Pinging 10.38.0.10 with 32 bytes of data:
Reply from 10.38.0.10: bytes=32 time=1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Ping statistics for 10.38.0.10:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms
The ODA will not permit to create any untagged additional network on the btbond2 interface knowing one tagged network already exists :
[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bond DCS-10001:Internal error encountered: Creating non-VLAN typed network on the interface btbond2 is not allowed. VLAN tagged38 already exists in interface btbond2.
Let’s create the second tagged network on the same btbond2 interface :
[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged39 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39
{
"jobId" : "e0452663-544c-47c4-8b9b-5fbe6e0a5cd9",
"status" : "Created",
"message" : null,
"reports" : [ ],
"createTimestamp" : "February 04, 2022 14:45:18 PM CET",
"resourceList" : [ ],
"description" : "Rac Network service creation with name tagged39",
"updatedTime" : "February 04, 2022 14:45:18 PM CET"
}
[root@dbi-oda-x8 ~]# odacli describe-job -i e0452663-544c-47c4-8b9b-5fbe6e0a5cd9
Job details
----------------------------------------------------------------
ID: e0452663-544c-47c4-8b9b-5fbe6e0a5cd9
Description: Rac Network service creation with name tagged39
Status: Success
Created: February 4, 2022 2:45:18 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Setting network February 4, 2022 2:45:18 PM CET February 4, 2022 2:45:23 PM CET Success
Setting up Vlan February 4, 2022 2:45:18 PM CET February 4, 2022 2:45:23 PM CET Success
New tagged network has been created and we now have 2 tagged network running on the same btbond2 interface :
[root@dbi-oda-x8 ~]# odacli list-networks ID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks -------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- ----------------------- bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241] 95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2] e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24] fbe05bfa-636e-4f9c-a348-c59ba23e2296 tagged38 btbond2.38 VLAN 255.255.255.0 10.38.0.1 38 [IP Address on node0: 10.38.0.10] dc11ecbe-c4be-4e18-9c37-9f6360b37ee1 tagged39 btbond2.39 VLAN 255.255.255.0 10.39.0.1 39 [IP Address on node0: 10.39.0.10]
The respective new tagged interface has been configured by odacli on the linux operating system :
[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2* -rw-r--r--. 1 root root 239 Feb 4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2 -rw-r--r-- 1 root root 520 Feb 4 14:40 /etc/sysconfig/network-scripts/ifcfg-btbond2.38 -rw-r--r-- 1 root root 520 Feb 4 14:45 /etc/sysconfig/network-scripts/ifcfg-btbond2.39 [root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.39 #ODA_VLAN_CONFIG === #ODA_VLAN_CONFIG Name=tagged39 #ODA_VLAN_CONFIG VlanId=39 #ODA_VLAN_CONFIG VlanInterface=btbond2 #ODA_VLAN_CONFIG Type=VlanType #ODA_VLAN_CONFIG VlanSetupType=Other #ODA_VLAN_CONFIG VlanIpAddr=10.39.0.10 #ODA_VLAN_CONFIG VlanNetmask=255.255.255.0 #ODA_VLAN_CONFIG VlanGateway=10.39.0.1 #ODA_VLAN_CONFIG NodeNum=0 #=== DO NOT EDIT ANYTHING ABOVE THIS LINE === DEVICE=btbond2.39 BOOTPROTO=none ONBOOT=yes VLAN=yes NM_CONTROLLED=no DEFROUTE=no IPADDR=10.39.0.10 NETMASK=255.255.255.0 GATEWAY=10.39.0.1
And I can ping my new network as well :
C:Users>ping 10.39.0.10
Pinging 10.39.0.10 with 32 bytes of data:
Reply from 10.39.0.10: bytes=32 time=2ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Ping statistics for 10.39.0.10:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 2ms, Average = 0ms
I can now as well check my global btbond2 configuration on the linux side :
[root@dbi-oda-x8 ~]# ip addr sh | grep -iE "(btbond2|btbond2.38|btbond2.39)"
4: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
9: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000
57: btbond2.38@btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.38.0.10/24 brd 10.38.0.255 scope global btbond2.38
58: btbond2.39@btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.39.0.10/24 brd 10.39.0.255 scope global btbond2.39
[root@dbi-oda-x8 ~]#
Creating virtual networks
Knowing I need to use the networks for the next DB Systems I have deleted the created physical networks and I’m going to create the 2 same tagged networks as virtual networks on the btbond2 interface :
[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged38 -if btbond2 -t bridged-vlan -ip 10.38.0.10 -nm 255.255.255.0 -vlan 38 -gw 10.38.0.1
Job details
----------------------------------------------------------------
ID: eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1
Description: vNetwork tagged38 creation
Status: Created
Created: February 4, 2022 3:19:02 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1
Job details
----------------------------------------------------------------
ID: eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1
Description: vNetwork tagged38 creation
Status: Success
Created: February 4, 2022 3:19:02 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network doesn‘t exist February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET Success
Validate interface to use exists February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET Success
Validate interfaces to create not exist February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET Success
Create bridge February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET Success
Create VLAN February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET Success
Bring up VLAN February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:07 PM CET Success
Create metadata February 4, 2022 3:19:07 PM CET February 4, 2022 3:19:07 PM CET Success
Persist metadata February 4, 2022 3:19:07 PM CET February 4, 2022 3:19:07 PM CET Success
[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -ip 10.39.0.10 -nm 255.255.255.0 -vlan 39 -gw 10.39.0.1
Job details
----------------------------------------------------------------
ID: 15cf889f-9c6e-4e0c-a676-b2203e40cfd2
Description: vNetwork tagged39 creation
Status: Created
Created: February 4, 2022 3:19:40 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i 15cf889f-9c6e-4e0c-a676-b2203e40cfd2
Job details
----------------------------------------------------------------
ID: 15cf889f-9c6e-4e0c-a676-b2203e40cfd2
Description: vNetwork tagged39 creation
Status: Success
Created: February 4, 2022 3:19:40 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network doesn‘t exist February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET Success
Validate interface to use exists February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET Success
Validate interfaces to create not exist February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET Success
Create bridge February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET Success
Create VLAN February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET Success
Bring up VLAN February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:45 PM CET Success
Create metadata February 4, 2022 3:19:45 PM CET February 4, 2022 3:19:45 PM CET Success
Persist metadata February 4, 2022 3:19:45 PM CET February 4, 2022 3:19:45 PM CET Success
My 2 new tagged networks exists as virtual network and not physical networks :
[root@dbi-oda-x8 ~]# odacli list-networks ID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks -------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- ----------------------- bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241] 95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2] e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24] [root@dbi-oda-x8 ~]# odacli list-vnetworks Name Type Interface Bridge Uniform Created Updated -------------------- --------------- --------------- -------------------- -------- ----------------------- ----------------------- tagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-04 15:19:45 CET 2022-02-04 15:19:45 CET tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CET pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
Tagged network interfaces have been created by the dcs-agent on the linux operating side :
[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2* -rw-r--r--. 1 root root 239 Feb 4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2 -rw-r--r-- 1 root root 145 Feb 4 15:19 /etc/sysconfig/network-scripts/ifcfg-btbond2.38 -rw-r--r-- 1 root root 145 Feb 4 15:19 /etc/sysconfig/network-scripts/ifcfg-btbond2.39 [root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.38 #This file was created by ODA. Do not edit. DEVICE=btbond2.38 ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=none VLAN=yes ONPARENT=yes BRIDGE=brtagged38 [root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.39 #This file was created by ODA. Do not edit. DEVICE=btbond2.39 ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=none VLAN=yes ONPARENT=yes BRIDGE=brtagged39
On the linux side, I can see that the respective IP addresses have not been assigned to the physical interfaces :
[root@dbi-oda-x8 ~]# ip addr sh | grep -iE "(btbond2|btbond2.38|btbond2.39)" 4: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000 5: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000 9: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000 60: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000 62: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000
But on new virtual interfaces linked to the tagged btbond2 interfaces :
[root@dbi-oda-x8 ~]# ip addr sh | grep -iE "(tagged38|tagged39)"
59: brtagged38: mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.38.0.10/24 brd 10.38.0.255 scope global brtagged38
60: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
61: brtagged39: mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.39.0.10/24 brd 10.39.0.255 scope global brtagged39
62: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000
And I can ping them :
C:Users>ping 10.38.0.10
Pinging 10.38.0.10 with 32 bytes of data:
Reply from 10.38.0.10: bytes=32 time=2ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
Reply from 10.38.0.10: bytes=32 time<1ms TTL=63
C:Users>ping 10.39.0.10
Pinging 10.39.0.10 with 32 bytes of data:
Reply from 10.39.0.10: bytes=32 time=2ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Reply from 10.39.0.10: bytes=32 time=1ms TTL=63
Reply from 10.39.0.10: bytes=32 time<1ms TTL=63
Ping statistics for 10.39.0.10:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 2ms, Average = 0ms
I can see the network bridge information from the operating system as well :
[root@dbi-oda-x8 ~]# brctl show bridge name bridge id STP enabled interfaces brtagged38 8000.5254003bdc76 no btbond2.38 vnet2 brtagged39 8000.52540060eb57 no btbond2.39 vnet4 privasm 8000.5a8241cdb508 no priv0.100 vnet1 vnet3 pubnet 8000.3cfdfe928018 no btbond1 vnet0 virbr0 8000.525400dc8c09 yes virbr0-nic
Creating 2 DB Systems using respectively tagged38 and tagged39 virtual networks
My repository has already been updated with the KVM DB System Image :
[root@dbi-oda-x8 ~]# odacli describe-dbsystem-image
DB System Image details
--------------------------------------------------------------------------------
Component Name Supported Versions Available Versions
-------------------- -------------------- --------------------
DBVM 19.13.0.0.0 19.13.0.0.0
GI 19.13.0.0.211019 19.13.0.0.211019
19.12.0.0.210720 not-available
19.11.0.0.210420 not-available
21.4.0.0.211019 not-available
21.3.0.0.210720 not-available
DB 19.13.0.0.211019 19.13.0.0.211019
19.12.0.0.210720 not-available
19.11.0.0.210420 not-available
21.4.0.0.211019 not-available
21.3.0.0.210720 not-available
I have created the first DB System json file assigning a new IP address from the VLAN38 network using tagged38 virtual network. The IP connection will use bridging through the virtual VLAN38 IP address we created on the Bare Metal itself :
[root@dbi-oda-x8 ~]# cat /opt/dbi/create_dbsystem_srvdb38.json
...
...
...
"network": {
"domainName": "dbi-lab.ch",
"ntpServers": ["216.239.35.0"],
"dnsServers": [
"8.8.8.8","8.8.4.4"
],
"nodes": [
{
"name": "srvdb38",
"ipAddress": "10.38.0.20",
"netmask": "255.255.255.0",
"gateway": "10.38.0.1",
"number": 0
}
],
"publicVNetwork": "tagged38"
},
"grid": {
"language": "en"
}
}
Creation of the first DB System on network VLAN 38 :
[root@dbi-oda-x8 ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb38.json
Enter password for system "srvdb38":
Retype password for system "srvdb38":
Enter administrator password for DB "DB38":
Retype administrator password for DB "DB38":
Job details
----------------------------------------------------------------
ID: ed88ef81-5cb3-4214-ac5c-bc255b67577f
Description: DB System srvdb38 creation
Status: Created
Created: February 4, 2022 4:12:33 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i ed88ef81-5cb3-4214-ac5c-bc255b67577f
Job details
----------------------------------------------------------------
ID: ed88ef81-5cb3-4214-ac5c-bc255b67577f
Description: DB System srvdb38 creation
Status: Success
Created: February 4, 2022 4:12:33 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create DB System metadata February 4, 2022 4:12:33 PM CET February 4, 2022 4:12:33 PM CET Success
Persist new DB System February 4, 2022 4:12:33 PM CET February 4, 2022 4:12:33 PM CET Success
Validate DB System prerequisites February 4, 2022 4:12:33 PM CET February 4, 2022 4:12:37 PM CET Success
Setup DB System environment February 4, 2022 4:12:37 PM CET February 4, 2022 4:12:39 PM CET Success
Create DB System ASM volume February 4, 2022 4:12:39 PM CET February 4, 2022 4:12:45 PM CET Success
Create DB System ACFS filesystem February 4, 2022 4:12:45 PM CET February 4, 2022 4:12:54 PM CET Success
Create DB System VM ACFS snapshots February 4, 2022 4:12:54 PM CET February 4, 2022 4:13:27 PM CET Success
Create temporary SSH key pair February 4, 2022 4:13:27 PM CET February 4, 2022 4:13:27 PM CET Success
Create DB System cloud-init config February 4, 2022 4:13:27 PM CET February 4, 2022 4:13:27 PM CET Success
Provision DB System VM(s) February 4, 2022 4:13:27 PM CET February 4, 2022 4:13:28 PM CET Success
Attach disks to DB System February 4, 2022 4:13:28 PM CET February 4, 2022 4:13:29 PM CET Success
Add DB System to Clusterware February 4, 2022 4:13:29 PM CET February 4, 2022 4:13:29 PM CET Success
Start DB System February 4, 2022 4:13:29 PM CET February 4, 2022 4:13:30 PM CET Success
Wait DB System VM first boot February 4, 2022 4:13:30 PM CET February 4, 2022 4:17:06 PM CET Success
Setup Mutual TLS (mTLS) February 4, 2022 4:17:06 PM CET February 4, 2022 4:21:36 PM CET Success
Export clones repository February 4, 2022 4:21:36 PM CET February 4, 2022 4:21:37 PM CET Success
Setup ASM client cluster config February 4, 2022 4:21:37 PM CET February 4, 2022 4:22:00 PM CET Success
Install DB System February 4, 2022 4:22:00 PM CET February 4, 2022 5:07:11 PM CET Success
Cleanup temporary SSH key pair February 4, 2022 5:07:11 PM CET February 4, 2022 5:07:52 PM CET Success
Set DB System as configured February 4, 2022 5:07:52 PM CET February 4, 2022 5:07:52 PM CET Success
I have then created the second DB System json file assigning a new IP address from the VLAN39 network using tagged39 virtual network. That time the IP connection will use bridging through the virtual VLAN39 IP address we created on the Bare Metal itself :
[root@dbi-oda-x8 ~]# cat /opt/dbi/create_dbsystem_srvdb39.json
...
...
...
},
"network": {
"domainName": "dbi-lab.ch",
"ntpServers": ["216.239.35.0"],
"dnsServers": [
"8.8.8.8","8.8.4.4"
],
"nodes": [
{
"name": "srvdb39",
"ipAddress": "10.39.0.20",
"netmask": "255.255.255.0",
"gateway": "10.39.0.1",
"number": 0
}
],
"publicVNetwork": "tagged39"
},
"grid": {
"language": "en"
}
}
Creation on the second DB System on network VLAN 39 :
[root@dbi-oda-x8 ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb39.json
Enter password for system "srvdb39":
Retype password for system "srvdb39":
Enter administrator password for DB "DB39":
Retype administrator password for DB "DB39":
Job details
----------------------------------------------------------------
ID: 38303453-8524-4c65-b1d2-3717dfc79a1f
Description: DB System srvdb39 creation
Status: Created
Created: February 4, 2022 5:14:01 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i 38303453-8524-4c65-b1d2-3717dfc79a1f
Job details
----------------------------------------------------------------
ID: 38303453-8524-4c65-b1d2-3717dfc79a1f
Description: DB System srvdb39 creation
Status: Success
Created: February 4, 2022 5:14:01 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create DB System metadata February 4, 2022 5:14:01 PM CET February 4, 2022 5:14:01 PM CET Success
Persist new DB System February 4, 2022 5:14:01 PM CET February 4, 2022 5:14:01 PM CET Success
Validate DB System prerequisites February 4, 2022 5:14:01 PM CET February 4, 2022 5:14:05 PM CET Success
Setup DB System environment February 4, 2022 5:14:05 PM CET February 4, 2022 5:14:07 PM CET Success
Create DB System ASM volume February 4, 2022 5:14:07 PM CET February 4, 2022 5:14:13 PM CET Success
Create DB System ACFS filesystem February 4, 2022 5:14:13 PM CET February 4, 2022 5:14:22 PM CET Success
Create DB System VM ACFS snapshots February 4, 2022 5:14:22 PM CET February 4, 2022 5:14:52 PM CET Success
Create temporary SSH key pair February 4, 2022 5:14:52 PM CET February 4, 2022 5:14:52 PM CET Success
Create DB System cloud-init config February 4, 2022 5:14:52 PM CET February 4, 2022 5:14:53 PM CET Success
Provision DB System VM(s) February 4, 2022 5:14:53 PM CET February 4, 2022 5:14:54 PM CET Success
Attach disks to DB System February 4, 2022 5:14:54 PM CET February 4, 2022 5:14:54 PM CET Success
Add DB System to Clusterware February 4, 2022 5:14:54 PM CET February 4, 2022 5:14:54 PM CET Success
Start DB System February 4, 2022 5:14:54 PM CET February 4, 2022 5:14:56 PM CET Success
Wait DB System VM first boot February 4, 2022 5:14:56 PM CET February 4, 2022 5:18:32 PM CET Success
Setup Mutual TLS (mTLS) February 4, 2022 5:18:32 PM CET February 4, 2022 5:23:07 PM CET Success
Export clones repository February 4, 2022 5:23:07 PM CET February 4, 2022 5:23:07 PM CET Success
Setup ASM client cluster config February 4, 2022 5:23:07 PM CET February 4, 2022 5:23:30 PM CET Success
Install DB System February 4, 2022 5:23:30 PM CET February 4, 2022 6:09:43 PM CET Success
Cleanup temporary SSH key pair February 4, 2022 6:09:43 PM CET February 4, 2022 6:10:23 PM CET Success
Set DB System as configured February 4, 2022 6:10:23 PM CET February 4, 2022 6:10:23 PM CET Success
Virtual network configuration checks
So I have got my both virtual networks configured on respective IP address (10.38.0.10 for VLAN38 network and 10.39.0.10 for VLAN39 network) :
[root@dbi-oda-x8 ~]# odacli list-vnetworks
Name Type Interface Bridge Uniform Created Updated
-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------
tagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-04 15:19:45 CET 2022-02-04 15:19:45 CET
tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CET
pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged38
VNetwork details
--------------------------------------------------------------------------------
ID: 0d8099ba-ff42-4d73-82b4-f7497fb68ca5
Name: tagged38
Created: 2022-02-04 15:19:07 CET
Updated: 2022-02-04 15:19:07 CET
Type: BridgedVlan
Interface name: btbond2
Bridge name: brtagged38
VLAN ID: 38
IP: 10.38.0.10
Netmask: 255.255.255.0
Gateway: 10.38.0.1
Attached in VMs (config): xa287c764d
Attached in VMs (live): xa287c764d
[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged39
VNetwork details
--------------------------------------------------------------------------------
ID: 00df04d4-9a01-4547-9a3c-5a27dab10494
Name: tagged39
Created: 2022-02-04 15:19:45 CET
Updated: 2022-02-04 15:19:45 CET
Type: BridgedVlan
Interface name: btbond2
Bridge name: brtagged39
VLAN ID: 39
IP: 10.39.0.10
Netmask: 255.255.255.0
Gateway: 10.39.0.1
Attached in VMs (config): x793d6c5ce
Attached in VMs (live): x793d6c5ce
DB System information checks
I have my 2 DB Systems.
srvdb38 configured on VLAN38 network with 10.38.0.20 IP Address.
srvdb39 configured on VLAN39 network with 10.39.0.20 IP Address.
[root@dbi-oda-x8 ~]# odacli list-dbsystems
Name Shape Cores Memory GI version DB version Status Created Updated
-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ----------------------- -----------------------
srvdb39 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 17:14:01 CET 2022-02-04 18:10:23 CET
srvdb38 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 16:12:33 CET 2022-02-04 17:07:52 CET
[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb38
DB System details
--------------------------------------------------------------------------------
ID: 859466fe-a4a5-40ae-aa00-8e02cf74dedc
Name: srvdb38
Image: 19.13.0.0.0
Shape: odb2
Cluster name: dbsa287c764d
Grid version: 19.13.0.0.211019
Memory: 16.00 GB
NUMA enabled: YES
Status: CONFIGURED
Created: 2022-02-04 16:12:33 CET
Updated: 2022-02-04 17:07:52 CET
CPU Pool
--------------------------
Name: cpupool4dbsystems
Number of cores: 4
Host: dbi-oda-x8
Effective CPU set: 5-6,21-22,37-38,53-54
Online CPUs: 5, 6, 21, 22, 37, 38, 53, 54
Offline CPUs: NONE
VM Storage
--------------------------
Disk group: DATA
Volume name: SA287C764D
Volume device: /dev/asm/sa287c764d-390
Size: 200.00 GB
Mount Point: /u05/app/sharedrepo/srvdb38
VMs
--------------------------
Host: dbi-oda-x8
VM Name: xa287c764d
VM Host Name: srvdb38.dbi-lab.ch
VM image path: /u05/app/sharedrepo/srvdb38/.ACFS/snaps/vm_xa287c764d/xa287c764d
Target State: ONLINE
Current State: ONLINE
VNetworks
--------------------------
Host: dbi-oda-x8
VM Name: xa287c764d
Public: 10.38.0.20 / 255.255.255.0 / ens3 / BRIDGE(brtagged38)
ASM: 192.168.17.4 / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)
Extra VNetworks
--------------------------
Host: dbi-oda-x8
VM Name: xa287c764d
tagged38: 10.38.0.20 / 255.255.255.0 / PUBLIC
Databases
--------------------------
Name: DB38
Resource ID: 5ff22777-f24b-446a-8c58-3fa6dd2ad383
Unique name: DB38_SITE1
Database ID: 715007309
Domain name: dbi-lab.ch
DB Home ID: 8b66a7d6-9e5e-43c0-a69c-00b80d2dba81
Shape: odb2
Version: 19.13.0.0.211019
Edition: EE
Type: SI
Role: PRIMARY
Class: OLTP
Storage: ASM
Redundancy:
Target node name:
Character set: AL32UTF8
NLS character set:
Language: ENGLISH
Territory: AMERICA
Console enabled: false
SEHA enabled: false
Associated networks: Public-network
Backup config ID:
Level 0 Backup Day: sunday
Autobackup enabled: true
TDE enabled: false
CDB type: false
PDB name:
PDB admin user:
[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb39
DB System details
--------------------------------------------------------------------------------
ID: 5238f074-6126-4051-8b9e-e392b55a2328
Name: srvdb39
Image: 19.13.0.0.0
Shape: odb2
Cluster name: dbs793d6c5ce
Grid version: 19.13.0.0.211019
Memory: 16.00 GB
NUMA enabled: YES
Status: CONFIGURED
Created: 2022-02-04 17:14:01 CET
Updated: 2022-02-04 18:10:23 CET
CPU Pool
--------------------------
Name: cpupool4dbsystems
Number of cores: 4
Host: dbi-oda-x8
Effective CPU set: 5-6,21-22,37-38,53-54
Online CPUs: 5, 6, 21, 22, 37, 38, 53, 54
Offline CPUs: NONE
VM Storage
--------------------------
Disk group: DATA
Volume name: S793D6C5CE
Volume device: /dev/asm/s793d6c5ce-390
Size: 200.00 GB
Mount Point: /u05/app/sharedrepo/srvdb39
VMs
--------------------------
Host: dbi-oda-x8
VM Name: x793d6c5ce
VM Host Name: srvdb39.dbi-lab.ch
VM image path: /u05/app/sharedrepo/srvdb39/.ACFS/snaps/vm_x793d6c5ce/x793d6c5ce
Target State: ONLINE
Current State: ONLINE
VNetworks
--------------------------
Host: dbi-oda-x8
VM Name: x793d6c5ce
Public: 10.39.0.20 / 255.255.255.0 / ens3 / BRIDGE(brtagged39)
ASM: 192.168.17.5 / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)
Extra VNetworks
--------------------------
Host: dbi-oda-x8
VM Name: x793d6c5ce
tagged39: 10.39.0.20 / 255.255.255.0 / PUBLIC
Databases
--------------------------
Name: DB39
Resource ID: e796e7e1-c8c8-4a63-809c-57976ce2163d
Unique name: DB39_SITE1
Database ID: 2043690133
Domain name: dbi-lab.ch
DB Home ID: 67f6df00-83d8-477b-8884-201664a3701b
Shape: odb2
Version: 19.13.0.0.211019
Edition: EE
Type: SI
Role: PRIMARY
Class: OLTP
Storage: ASM
Redundancy:
Target node name:
Character set: AL32UTF8
NLS character set:
Language: ENGLISH
Territory: AMERICA
Console enabled: false
SEHA enabled: false
Associated networks: Public-network
Backup config ID:
Level 0 Backup Day: sunday
Autobackup enabled: true
TDE enabled: false
CDB type: false
PDB name:
PDB admin user:
[root@dbi-oda-x8 ~]#
Direct connection on the DB Systems
Finally I can access directly the DB Systems through SSH from my laptop. Here as example, connection to the srvdb38 DB System.
maw@DBI-LT-MAW2 ~ % ssh [email protected] [email protected]'s password: Last login: Thu Feb 17 16:07:31 2022 [root@srvdb38 ~]# ip addr sh ens3 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:6e:3f:9f brd ff:ff:ff:ff:ff:ff inet 10.38.0.20/24 brd 10.38.0.255 scope global noprefixroute ens3 valid_lft forever preferred_lft forever [root@srvdb38 ~]# ps -ef | grep [p]mon oracle 21506 1 0 14:05 ? 00:00:00 ora_pmon_DB38 [root@srvdb38 ~]#
And directly from my laptop I can connect and use the Oracle database DB38 :
Creating a virtual network without assigning an IP address to the bridge interface
Of course it is possible to create a virtual bridge interface without assigning any IP address.
I deleted tagged39 virtual network in order to create it again without IP address :
[root@dbi-oda-x8 ~]# odacli list-vnetworks
Name Type Interface Bridge Uniform Created Updated
-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------
tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CET
pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
tagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-04 22:13:18 CET 2022-02-17 16:38:02 CET
[root@dbi-oda-x8 ~]# odacli delete-vnetwork -n tagged39
Job details
----------------------------------------------------------------
ID: 2cae44c4-39e7-4102-8453-597f707ed10b
Description: vNetwork tagged39 deletion
Status: Created
Created: February 17, 2022 4:46:45 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i 2cae44c4-39e7-4102-8453-597f707ed10b
Job details
----------------------------------------------------------------
ID: 2cae44c4-39e7-4102-8453-597f707ed10b
Description: vNetwork tagged39 deletion
Status: Success
Created: February 17, 2022 4:46:45 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network exists February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:45 PM CET Success
Validate VM does not have the VNetwork attached February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:45 PM CET Success
Delete bridge February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:45 PM CET Success
Delete VLAN February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:46 PM CET Success
Delete metadata February 17, 2022 4:46:46 PM CET February 17, 2022 4:46:46 PM CET Success
[root@dbi-oda-x8 ~]# odacli list-vnetworks
Name Type Interface Bridge Uniform Created Updated
-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------
tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CET
pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
[root@dbi-oda-x8 ~]# ip addr sh | grep btbond2
4: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
10: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000
11: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
[root@dbi-oda-x8 ~]#
Of course I could have updated the vnetwork interface using odacli modify-vnetwork and assigning 0.0.0.0 as IP address.
I created again the tagged39 virtual network without assigning an IP address to the bridge interface.
[root@dbi-oda-x8 ~]# odacli list-vnetworks
Name Type Interface Bridge Uniform Created Updated
-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------
tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CET
pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -vlan 39
Job details
----------------------------------------------------------------
ID: a70a1313-f2cd-48ab-ac5f-c17b2f2fae08
Description: vNetwork tagged39 creation
Status: Created
Created: February 17, 2022 4:49:44 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i a70a1313-f2cd-48ab-ac5f-c17b2f2fae08
Job details
----------------------------------------------------------------
ID: a70a1313-f2cd-48ab-ac5f-c17b2f2fae08
Description: vNetwork tagged39 creation
Status: Success
Created: February 17, 2022 4:49:44 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate Virtual Network doesn't exist February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET Success
Validate interface to use exists February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET Success
Validate interfaces to create not exist February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET Success
Create bridge February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET Success
Create VLAN February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET Success
Bring up VLAN February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:45 PM CET Success
Create metadata February 17, 2022 4:49:45 PM CET February 17, 2022 4:49:45 PM CET Success
Persist metadata February 17, 2022 4:49:45 PM CET February 17, 2022 4:49:45 PM CET Success
[root@dbi-oda-x8 ~]# odacli list-vnetworks
Name Type Interface Bridge Uniform Created Updated
-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------
tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CET
pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET
tagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-17 16:49:45 CET 2022-02-17 16:49:45 CET
[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged39
VNetwork details
--------------------------------------------------------------------------------
ID: 6e40ca21-1d4a-4bcc-8436-f29ffb2eb8bc
Name: tagged39
Created: 2022-02-17 16:49:45 CET
Updated: 2022-02-17 16:49:45 CET
Type: BridgedVlan
Interface name: btbond2
Bridge name: brtagged39
VLAN ID: 39
IP:
Netmask:
Gateway:
Attached in VMs (config): NONE
Attached in VMs (live): NONE
[root@dbi-oda-x8 ~]# ip addr sh | grep btbond2
4: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
5: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 1000
10: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000
11: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
31: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000
[root@dbi-oda-x8 ~]# ip addr sh | grep brtagged
11: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000
12: brtagged38: mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.38.0.10/24 brd 10.38.0.255 scope global brtagged38
25: vnet6: mtu 1500 qdisc pfifo_fast master brtagged38 state UNKNOWN group default qlen 1000
30: brtagged39: mtu 1500 qdisc noqueue state UP group default qlen 1000
31: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000
[root@dbi-oda-x8 ~]#
And let’s deploy the srvdb39 DB Systems again after having removed it with odacli delete-dbsystem command :
[root@dbi-oda-x8 ~]# odacli list-dbsystems
Name Shape Cores Memory GI version DB version Status Created Updated
-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ----------------------- -----------------------
srvdb38 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 16:12:33 CET 2022-02-04 17:07:52 CET
[root@dbi-oda-x8 ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb39.json
Enter password for system "srvdb39":
Retype password for system "srvdb39":
Enter administrator password for DB "DB39":
Retype administrator password for DB "DB39":
Job details
----------------------------------------------------------------
ID: 934c752b-7eb6-4b5b-8856-7a90102a1e65
Description: DB System srvdb39 creation
Status: Created
Created: February 17, 2022 5:06:37 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[root@dbi-oda-x8 ~]# odacli describe-job -i 934c752b-7eb6-4b5b-8856-7a90102a1e65
Job details
----------------------------------------------------------------
ID: 934c752b-7eb6-4b5b-8856-7a90102a1e65
Description: DB System srvdb39 creation
Status: Success
Created: February 17, 2022 5:06:37 PM CET
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Create DB System metadata February 17, 2022 5:06:37 PM CET February 17, 2022 5:06:37 PM CET Success
Persist new DB System February 17, 2022 5:06:37 PM CET February 17, 2022 5:06:37 PM CET Success
Validate DB System prerequisites February 17, 2022 5:06:37 PM CET February 17, 2022 5:06:41 PM CET Success
Setup DB System environment February 17, 2022 5:06:41 PM CET February 17, 2022 5:06:43 PM CET Success
Create DB System ASM volume February 17, 2022 5:06:43 PM CET February 17, 2022 5:06:49 PM CET Success
Create DB System ACFS filesystem February 17, 2022 5:06:49 PM CET February 17, 2022 5:06:58 PM CET Success
Create DB System VM ACFS snapshots February 17, 2022 5:06:58 PM CET February 17, 2022 5:07:31 PM CET Success
Create temporary SSH key pair February 17, 2022 5:07:31 PM CET February 17, 2022 5:07:32 PM CET Success
Create DB System cloud-init config February 17, 2022 5:07:32 PM CET February 17, 2022 5:07:32 PM CET Success
Provision DB System VM(s) February 17, 2022 5:07:32 PM CET February 17, 2022 5:07:33 PM CET Success
Attach disks to DB System February 17, 2022 5:07:33 PM CET February 17, 2022 5:07:34 PM CET Success
Add DB System to Clusterware February 17, 2022 5:07:34 PM CET February 17, 2022 5:07:34 PM CET Success
Start DB System February 17, 2022 5:07:34 PM CET February 17, 2022 5:07:36 PM CET Success
Wait DB System VM first boot February 17, 2022 5:07:36 PM CET February 17, 2022 5:08:48 PM CET Success
Setup Mutual TLS (mTLS) February 17, 2022 5:08:48 PM CET February 17, 2022 5:09:08 PM CET Success
Export clones repository February 17, 2022 5:09:08 PM CET February 17, 2022 5:09:08 PM CET Success
Setup ASM client cluster config February 17, 2022 5:09:08 PM CET February 17, 2022 5:09:11 PM CET Success
Install DB System February 17, 2022 5:09:11 PM CET February 17, 2022 5:36:57 PM CET Success
Cleanup temporary SSH key pair February 17, 2022 5:36:57 PM CET February 17, 2022 5:36:57 PM CET Success
Set DB System as configured February 17, 2022 5:36:57 PM CET February 17, 2022 5:36:57 PM CET Success
[root@dbi-oda-x8 ~]# odacli list-dbsystems
Name Shape Cores Memory GI version DB version Status Created Updated
-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ----------------------- -----------------------
srvdb38 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 16:12:33 CET 2022-02-04 17:07:52 CET
srvdb39 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-17 17:06:37 CET 2022-02-17 17:36:57 CET
[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb39
DB System details
--------------------------------------------------------------------------------
ID: ed13bacb-be28-4e31-a5d7-39fd713d72a0
Name: srvdb39
Image: 19.13.0.0.0
Shape: odb2
Cluster name: dbs793d6c5ce
Grid version: 19.13.0.0.211019
Memory: 16.00 GB
NUMA enabled: YES
Status: CONFIGURED
Created: 2022-02-17 17:06:37 CET
Updated: 2022-02-17 17:36:57 CET
CPU Pool
--------------------------
Name: cpupool4dbsystems
Number of cores: 4
Host: dbi-oda-x8
Effective CPU set: 5-6,21-22,37-38,53-54
Online CPUs: 5, 6, 21, 22, 37, 38, 53, 54
Offline CPUs: NONE
VM Storage
--------------------------
Disk group: DATA
Volume name: S793D6C5CE
Volume device: /dev/asm/s793d6c5ce-390
Size: 200.00 GB
Mount Point: /u05/app/sharedrepo/srvdb39
VMs
--------------------------
Host: dbi-oda-x8
VM Name: x793d6c5ce
VM Host Name: srvdb39.dbi-lab.ch
VM image path: /u05/app/sharedrepo/srvdb39/.ACFS/snaps/vm_x793d6c5ce/x793d6c5ce
Target State: ONLINE
Current State: ONLINE
VNetworks
--------------------------
Host: dbi-oda-x8
VM Name: x793d6c5ce
Public: 10.39.0.20 / 255.255.255.0 / ens3 / BRIDGE(brtagged39)
ASM: 192.168.17.5 / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100)
Extra VNetworks
--------------------------
Host: dbi-oda-x8
VM Name: x793d6c5ce
tagged39: 10.39.0.20 / 255.255.255.0 / PUBLIC
Databases
--------------------------
Name: DB39
Resource ID: 6ecee323-cd18-4971-acb4-b476d4d76d2c
Unique name: DB39_SITE1
Database ID: 2044811647
Domain name: dbi-lab.ch
DB Home ID: 3a6f8134-407a-4272-8512-7c52bdca68ea
Shape: odb2
Version: 19.13.0.0.211019
Edition: EE
Type: SI
Role: PRIMARY
Class: OLTP
Storage: ASM
Redundancy:
Target node name:
Character set: AL32UTF8
NLS character set:
Language: ENGLISH
Territory: AMERICA
Console enabled: false
SEHA enabled: false
Associated networks: Public-network
Backup config ID:
Level 0 Backup Day: sunday
Autobackup enabled: true
TDE enabled: false
CDB type: false
PDB name:
PDB admin user:
This DB System is also now accessible through his IP address 10.39.0.20 and is using the bridge interface without any IP address. A bridge interface does not need any IP address.
I can ping the new DB System from my laptop :
maw@DBI-LT-MAW2 ~ % ping 10.39.0.20 PING 10.39.0.20 (10.39.0.20): 56 data bytes 64 bytes from 10.39.0.20: icmp_seq=0 ttl=62 time=56.761 ms 64 bytes from 10.39.0.20: icmp_seq=1 ttl=62 time=56.734 ms 64 bytes from 10.39.0.20: icmp_seq=2 ttl=62 time=64.759 ms ^C --- 10.39.0.20 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 56.734/59.418/64.759/3.777 ms
I can ssh the new DB System from my laptop :
maw@DBI-LT-MAW2 ~ % ssh [email protected] [email protected]'s password: Last login: Thu Feb 17 18:10:10 2022 from gateway [root@srvdb39 ~]# ip addr sh ens3 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:86:87:88 brd ff:ff:ff:ff:ff:ff inet 10.39.0.20/24 brd 10.39.0.255 scope global noprefixroute ens3 valid_lft forever preferred_lft forever [root@srvdb39 ~]# ps -ef | grep [p]mon oracle 93401 1 0 17:35 ? 00:00:00 ora_pmon_DB39 [root@srvdb39 ~]#
And I can join the DB39 database from my laptop directly :
Conclusion
KVM DB System main purpose is hard partitioning for oracle licensing. All ODA’s cpu cores can remain active. DB System with appropriate pinned CPU core related to the number of licenses will be created for the oracle database. The others ODA’s CPU core can be used for further KVM or application needs.
As we could see KVM DB System will then have an additional pros. With DB System we can separate Database traffic through separated VLANs.
On the other hand KVM DB System will have some other cons :
- More complex installation.
- More resources will be consumed : each DB Systems will run its own oracle restart (grid) infrastructure.
- Each DB Systems will have to be patched separately as well during ODA patching, making patching process more heavy.
Also please note that current ODA release 19.13 supports only creation of database in version 19.11, 19.12, 19.13, 21.3 and 21.4 for the DB Systems.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MAW_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2024/04/SIT_web.png)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2023/05/AEA_web_min.jpg)
tagrub
13.06.2024Hello, once vnetwork are created for tagget vlan, type bridged-vlan, and dbsystems created on it , we would like to view these same vlan as network (odacli list-networks) for baremetal database creation on it via tooling. Seems not possible ? like it is for non-tagged pubnet seen as "bridge" on both sides ? technically as the bridge for tagged vlan can be itself assigned an ip, it is possible to add a database listener on it, but it is manual, outside tooling...
regards
Paul
Marc Wagner
24.06.2024Hi Paul,
Sorry for delay in my answer. You are right, creating vnetwork will not be listed in as network, and you can see this in my blog as well. It is different for the oda public interface (pubnet) that you will see in vnetwork list and network list. On the other hand the network you created as vnetwork is in any case also accessible from the bare metal if you assigned an IP address to your vnetwork. So you could create an additional listener on those ip address. Need to be tested on lab, I do not see any reason why it should not work. Please read my other article, https://www.dbi-services.com/blog/separating-database-traffic-on-an-oda/, where I explained all this. Of course creating this additional listener in the grid infra is manual. There is, so far, no odacli command which will do it for you.
Good luck.
Regards,
Marc