Introduction

If you use Enterprise Edition on your Oracle Database Appliance environment, and if you don’t have unlimited licences, you probably lowered the CPU cores available on each server to dispatch your licences accross your environment. This operation is normally done after deploying your ODAs, and this number of cores could eventually be increased if you need more and if you buy additional licences later. For most of us, the CPU core configuration will stay in its initial setup. But patching your ODA can enable back all the cores on your server, without any warning. Your configuration will then no more comply with your licences, and you may not even notice.

Use case

I recently patched several ODAs X8-2M from 19.11 to 19.14. After solving issues described in the pre-patch report, I was able to apply the system patch without any problem.

odacli describe-job -i 81c88506-5e05-4050-a244-b36d0c343327
Job details
----------------------------------------------------------------
                     ID:  81c88506-5e05-4050-a244-b36d0c343327
            Description:  Server Patching
                 Status:  Success
                Created:  August 17, 2022 9:13:32 AM CEST
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              August 17, 2022 9:13:42 AM CEST     August 17, 2022 9:13:42 AM CEST     Success
Validate ILOM server reachable           August 17, 2022 9:13:42 AM CEST     August 17, 2022 9:13:42 AM CEST     Success
Configure export clones resource         August 17, 2022 9:15:31 AM CEST     August 17, 2022 9:15:32 AM CEST     Success
Creating repositories using yum          August 17, 2022 9:15:32 AM CEST     August 17, 2022 9:15:36 AM CEST     Success
Updating YumPluginVersionLock rpm        August 17, 2022 9:15:36 AM CEST     August 17, 2022 9:15:36 AM CEST     Success
Applying OS Patches                      August 17, 2022 9:15:36 AM CEST     August 17, 2022 9:24:57 AM CEST     Success
Creating repositories using yum          August 17, 2022 9:24:57 AM CEST     August 17, 2022 9:24:57 AM CEST     Success
Applying HMP Patches                     August 17, 2022 9:24:57 AM CEST     August 17, 2022 9:25:14 AM CEST     Success
Patch location validation                August 17, 2022 9:25:14 AM CEST     August 17, 2022 9:25:14 AM CEST     Success
oda-hw-mgmt upgrade                      August 17, 2022 9:25:14 AM CEST     August 17, 2022 9:25:45 AM CEST     Success
OSS Patching                             August 17, 2022 9:25:45 AM CEST     August 17, 2022 9:25:46 AM CEST     Success
Applying Firmware Disk Patches           August 17, 2022 9:25:46 AM CEST     August 17, 2022 9:25:48 AM CEST     Success
Applying Firmware Controller Patches     August 17, 2022 9:25:48 AM CEST     August 17, 2022 9:25:50 AM CEST     Success
Checking Ilom patch Version              August 17, 2022 9:25:50 AM CEST     August 17, 2022 9:25:50 AM CEST     Success
Patch location validation                August 17, 2022 9:25:50 AM CEST     August 17, 2022 9:25:50 AM CEST     Success
Save password in Wallet                  August 17, 2022 9:25:50 AM CEST     August 17, 2022 9:25:51 AM CEST     Success
Disabling IPMI v2                        August 17, 2022 9:25:51 AM CEST     August 17, 2022 9:25:52 AM CEST     Success
Apply Ilom patch                         August 17, 2022 9:25:52 AM CEST     August 17, 2022 9:34:52 AM CEST     Success
Copying Flash Bios to Temp location      August 17, 2022 9:34:52 AM CEST     August 17, 2022 9:34:52 AM CEST     Success
Patch location validation                August 17, 2022 9:34:52 AM CEST     August 17, 2022 9:34:52 AM CEST     Success
ASR Manager RPM update                   August 17, 2022 9:34:52 AM CEST     August 17, 2022 9:49:30 AM CEST     Success
Modify JavaExec Path                     August 17, 2022 9:49:30 AM CEST     August 17, 2022 9:49:30 AM CEST     Success
Remove AsrConfBackup File                August 17, 2022 9:49:30 AM CEST     August 17, 2022 9:49:30 AM CEST     Success
Starting the clusterware                 August 17, 2022 9:49:31 AM CEST     August 17, 2022 9:53:15 AM CEST     Success
registering image                        August 17, 2022 9:53:15 AM CEST     August 17, 2022 9:53:16 AM CEST     Success
registering working copy                 August 17, 2022 9:53:16 AM CEST     August 17, 2022 9:53:16 AM CEST     Success
registering image                        August 17, 2022 9:53:16 AM CEST     August 17, 2022 9:53:16 AM CEST     Success
Creating GI home directories             August 17, 2022 9:53:16 AM CEST     August 17, 2022 9:53:16 AM CEST     Success
Extract GI clone                         August 17, 2022 9:53:16 AM CEST     August 17, 2022 9:53:16 AM CEST     Success
Provisioning Software Only GI with RHP   August 17, 2022 9:53:16 AM CEST     August 17, 2022 9:53:16 AM CEST     Success
Patch GI with RHP                        August 17, 2022 9:53:16 AM CEST     August 17, 2022 10:00:49 AM CEST    Success
Updating GIHome version                  August 17, 2022 10:00:49 AM CEST    August 17, 2022 10:00:52 AM CEST    Success
Validate GI availability                 August 17, 2022 10:01:04 AM CEST    August 17, 2022 10:01:04 AM CEST    Success
Patch KVM CRS type                       August 17, 2022 10:01:04 AM CEST    August 17, 2022 10:01:05 AM CEST    Success
Update System version                    August 17, 2022 10:01:05 AM CEST    August 17, 2022 10:01:05 AM CEST    Success
Cleanup JRE Home                         August 17, 2022 10:01:05 AM CEST    August 17, 2022 10:01:05 AM CEST    Success
Add SYSNAME in Env                       August 17, 2022 10:01:05 AM CEST    August 17, 2022 10:01:05 AM CEST    Success
Starting the clusterware                 August 17, 2022 10:01:05 AM CEST    August 17, 2022 10:01:05 AM CEST    Success
Setting ACL for disk groups              August 17, 2022 10:01:05 AM CEST    August 17, 2022 10:01:10 AM CEST    Success
Update lvm.conf file                     August 17, 2022 10:02:40 AM CEST    August 17, 2022 10:02:40 AM CEST    Success
Update previous workarounds              August 17, 2022 10:02:40 AM CEST    August 17, 2022 10:02:40 AM CEST    Success
preRebootNode Actions                    August 17, 2022 10:02:40 AM CEST    August 17, 2022 10:16:45 AM CEST    Success
Reboot Ilom                              August 17, 2022 10:16:45 AM CEST    August 17, 2022 10:16:45 AM CEST    Success

How to detect the problem?

After patching the server and after the reboot, all cores are enabled as if I never applied the core reduction:

odacli describe-cpucore
Node  Cores  Modified                           Job Status     
----- ------ ---------------------------------- ---------------
0     8      January 19, 2021 1:58:17 PM CET    CONFIGURED     

lscpu | grep CPU
CPU op-mode(s):        32-bit, 64-bit
CPU(s):                64
On-line CPU(s) list:   0-63
CPU family:            6
Model name:            Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
CPU MHz:               2800.135
CPU max MHz:           3900.0000
CPU min MHz:           1000.0000
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63

Threads from 0 to 63 are enabled, meaning 32 cores. This is the total number of cores of my two Xeon Gold 5218. But I only need 8 cores and paid for 8 cores, meaning 4 Enterprise Edition licences. My ODA is no more compliant with my licences after this patch.

How to fix the problem?

I will try to configure again 8 cores on my ODA using odacli:

odacli update-cpucore -c 8
Modifying the enabled number of CPU requires a reboot of all nodes in the ODA system. Are you sure you want to proceed with this operation? (Y/N): Y
DCS-10044:Object cpu cores already exists with same value 8.

It doesn’t work because 8 is my actual core configuration according to odacli.

I will decrease the number of cores, and then increase it again:

odacli update-cpucore -c 6 -f
Modifying the enabled number of CPU requires a reboot of all nodes in the ODA system. Are you sure you want to proceed with this operation? (Y/N): Y
{
  "jobId" : "c0e0befd-119b-4c96-9b88-cfa5b21e37b5",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "August 17, 2022 11:34:18 AM CEST",
  "resourceList" : [ ],
  "description" : "CPU cores  service update",
  "updatedTime" : "August 17, 2022 11:34:18 AM CEST"
}
WARNING: Reducing the number of enabled cores may set offline any of the CPUs used by a CPU Pool. Thus, please review the status of all the CPU Pools after this operation using the 'describe-cpupool' command. In case of any CPU from a CPU Pool was offline please run 'remap-cpupools -f' to update the list of CPUs and use only online CPUs.
WARNING: All the nodes in this ODA system will reboot after this operation has completed.

odacli update-cpucore -c 8
Modifying the enabled number of CPU requires a reboot of all nodes in the ODA system. Are you sure you want to proceed with this operation? (Y/N): Y
{
  "jobId" : "574a6b7f-ef91-4fd0-9202-1f977fad151b",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "August 17, 2022 11:47:23 AM CEST",
  "resourceList" : [ ],
  "description" : "CPU cores  service update",
  "updatedTime" : "August 17, 2022 11:47:23 AM CEST"
}
WARNING: Reducing the number of enabled cores may set offline any of the CPUs used by a CPU Pool. Thus, please review the status of all the CPU Pools after this operation using the 'describe-cpupool' command. In case of any CPU from a CPU Pool was offline please run 'remap-cpupools -f' to update the list of CPUs and use only online CPUs.
WARNING: All the nodes in this ODA system will reboot after this operation has completed.

odacli describe-cpucore
Node  Cores  Modified                           Job Status     
----- ------ ---------------------------------- ---------------
0     8      August 17, 2022 11:49:12 AM CEST   CONFIGURED     

lscpu | grep CPU
CPU op-mode(s):        32-bit, 64-bit
CPU(s):                16
On-line CPU(s) list:   0-15
CPU family:            6
Model name:            Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
CPU MHz:               3699.938
CPU max MHz:           3900.0000
CPU min MHz:           1000.0000
NUMA node0 CPU(s):     0-3,8-11
NUMA node1 CPU(s):     4-7,12-15

odacli describe-system | grep Core
         CPU Core Count: 8

My cores setup is back to normal.

Limits

This will not work if you only have 2 cores configured on your ODA. You will then need to increase to 4 cores, and decrease to 2 cores. This can be a problem as this temporary increase can be detected and it’s not allowed. I would recommend opening a Service Request on My Oracle Support. You’re not supposed to move CPU cores’ configuration if it’s not related to licences upgrade.

Conclusion

I don’t know if this problem is specific to this environment or if it’s more widespread. But you should add another post-patch check when applying a patch on your ODA: check with lscpu if the number of enabled cores matches your odacli configuration.

Post Views: 2,398
Thumbnail [60x60]
by
Jérôme Dubar