ODA update-server to 19.12 fails on Patch GI with RHP

Sometimes giving a workshop is a good opportunities to test new things…this what happened to me today…  🙂

While giving an ODA workshop, I thought it was a good opportunity to test patching to the new 19.12 release.
However during the update-server process we got a failure on the our ODA update to 19.12 failing on GI patch using RHP.

Let’s see how we fixed it…

So this is our starting point:

[root@dbi-oda-x8 tmp]# odacli describe-job -i d3efbbf5-f1db-4c5e-8f03-e8876f85a341

Job details
----------------------------------------------------------------
                     ID:  d3efbbf5-f1db-4c5e-8f03-e8876f85a341
            Description:  Server Patching
                 Status:  Failure
                Created:  November 4, 2021 11:01:15 AM CET
                Message:  DCS-10001:Internal error encountered: Fail to patch GI with RHP : DCS-10001:Internal error encountered: DCS-10001:Internal error encountered: clonemetadata.xml file is not present in local repository.
Update repository with new clones and retry same command...

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Server patching                          November 4, 2021 11:01:23 AM CET    November 4, 2021 11:22:16 AM CET    Failure
Server patching                          November 4, 2021 11:01:24 AM CET    November 4, 2021 11:22:16 AM CET    Failure
Creating repositories using yum          November 4, 2021 11:02:36 AM CET    November 4, 2021 11:02:38 AM CET    Success
Updating YumPluginVersionLock rpm        November 4, 2021 11:02:38 AM CET    November 4, 2021 11:02:38 AM CET    Success
Applying OS Patches                      November 4, 2021 11:02:38 AM CET    November 4, 2021 11:10:49 AM CET    Success
Creating repositories using yum          November 4, 2021 11:10:50 AM CET    November 4, 2021 11:10:50 AM CET    Success
Applying HMP Patches                     November 4, 2021 11:10:50 AM CET    November 4, 2021 11:11:08 AM CET    Success
Patch location validation                November 4, 2021 11:11:08 AM CET    November 4, 2021 11:11:08 AM CET    Success
oda-hw-mgmt upgrade                      November 4, 2021 11:11:09 AM CET    November 4, 2021 11:11:41 AM CET    Success
OSS Patching                             November 4, 2021 11:11:42 AM CET    November 4, 2021 11:11:42 AM CET    Success
Applying Firmware Disk Patches           November 4, 2021 11:11:42 AM CET    November 4, 2021 11:11:46 AM CET    Success
Applying Firmware Controller Patches     November 4, 2021 11:11:46 AM CET    November 4, 2021 11:11:50 AM CET    Success
Checking Ilom patch Version              November 4, 2021 11:11:50 AM CET    November 4, 2021 11:11:50 AM CET    Success
Patch location validation                November 4, 2021 11:11:50 AM CET    November 4, 2021 11:11:50 AM CET    Success
Save password in Wallet                  November 4, 2021 11:11:50 AM CET    November 4, 2021 11:11:51 AM CET    Success
Apply Ilom patch                         November 4, 2021 11:11:51 AM CET    November 4, 2021 11:20:48 AM CET    Success
Copying Flash Bios to Temp location      November 4, 2021 11:20:48 AM CET    November 4, 2021 11:20:48 AM CET    Success
Server patching                          November 4, 2021 11:20:48 AM CET    November 4, 2021 11:22:16 AM CET    Failure
Starting the clusterware                 November 4, 2021 11:20:48 AM CET    November 4, 2021 11:22:16 AM CET    Success
registering image                        November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
registering working copy                 November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
registering image                        November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Creating GI home directories             November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Extract GI clone                         November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Provisioning Software Only GI with RHP   November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Success
Patch GI with RHP                        November 4, 2021 11:22:16 AM CET    November 4, 2021 11:22:16 AM CET    Failure

So basically we had a failure because of

Patch GI with RHP November 4, 2021 11:22:16 AM CET November 4, 2021 11:22:16 AM CET Failure

with as explanation the following message:

DCS-10001:Internal error encountered: Fail to patch GI with RHP : DCS-10001:Internal error encountered:

DCS-10001:Internal error encountered: clonemetadata.xml file is not present in local repository.

This basically sounded like with we were missing the GI clone for the the 19.12 in our local repository (/opt/oracle/oak/pkgrepos)

The first check we was to verify one more time that our update repository job, to import the GI clone, worked properly

[root@dbi-oda-x8 clones]# odacli describe-job -i 3792c33b-3804-44f7-9806-eddf21bb4939

Job details
----------------------------------------------------------------
                     ID:  3792c33b-3804-44f7-9806-eddf21bb4939
            Description:  Repository Update
                 Status:  Success
                Created:  November 4, 2021 10:20:26 AM CET
                Message:  /tmp/ODA/odacli-dcs-19.12.0.0.0-210822.1-DB-19.12.0.0.zip,/tmp/ODA/odacli-dcs-19.12.0.0.0-210822.1-GI-19.12.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             November 4, 2021 10:20:29 AM CET    November 4, 2021 10:21:56 AM CET    Success
registering image                        November 4, 2021 10:21:57 AM CET    November 4, 2021 10:21:57 AM CET    Success
registering image                        November 4, 2021 10:21:57 AM CET    November 4, 2021 10:21:57 AM CET    Success

OK as this was fine, we decided to check if the file is really physically there:

[root@dbi-oda-x8 ~]# cd /opt/oracle/oak/pkgrepos/orapkgs/clones/
[root@dbi-oda-x8 clones]# ls -l
total 22095968
-rw-r--r-- 1 root root      13813 Aug 23 19:06 clonemetadata.xml
-rw-r--r-- 1 root root 4500047202 Feb  9  2021 db19.210119.tar.gz
-r-xr-xr-x 1 root root 5033359682 Aug 23 19:10 db19.210720.tar.gz
-r-xr-xr-x 1 root root 6542844964 Aug 23 19:06 grid19.210720.tar.gz
-rw-r--r-- 1 root root 6542464864 Feb  9  2021 grid19.tar.gz
drwx------ 2 root root      65536 Nov  4 11:01 lost+found

Hmmm the file is there and with a simple grep command we can confirm that it is available in the clonemedata.xml too.

We started searching in the ODA 19.12 Release Note for the known issues and the end of the list we found the following one:

Ok it is not exactly the same message (not talking about clonemedata.xml) but still linked to missing version.
The proposed workaround is:

So let’s have a look to our setup on the source GI Home

[root@dbi-oda-x8 bin]# ls -l osdbagrp*
-rwxr-xr-x 1 root oinstall 33488 Nov  3 13:24 osdbagrp
-rw-r----- 1 root oinstall     0 Feb  9  2021 osdbagrp0

As in the know issue, the file is not belonging to grid user. We tried then to change the ownership

[root@dbi-oda-x8 bin]# chown grid osdbagrp


[root@dbi-oda-x8 bin]# ls -l osdbagrp*
-rwxr-xr-x 1 grid oinstall 33488 Nov  3 13:24 osdbagrp
-rw-r----- 1 root oinstall     0 Feb  9  2021 osdbagrp0

Then we run the command recommended to update the registry…unfortunately neither worked

[root@dbi-oda-x8 bin]# odacli update-registry -n gihome
DCS-10112:Specified components are already discovered.


[root@dbi-oda-x8 bin]# odacli update-registry -n system
DCS-10112:Specified components are already discovered.

As it didn’t wanted to update the information, we decided to force the update

[root@dbi-oda-x8 bin]# odacli update-registry -n gihome -f

Job details
----------------------------------------------------------------
                     ID:  4a9c149b-d59a-4a05-baea-2acfa446ec34
            Description:  Discover System Components : gihome
                 Status:  Created
                Created:  November 4, 2021 11:42:05 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

Checking the state of the job shows that it is sucessful

[root@dbi-oda-x8 bin]# odacli describe-job -i 4a9c149b-d59a-4a05-baea-2acfa446ec34

Job details
----------------------------------------------------------------
                     ID:  4a9c149b-d59a-4a05-baea-2acfa446ec34
            Description:  Discover System Components : gihome
                 Status:  Success
                Created:  November 4, 2021 11:42:05 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Rediscover GiHome                        November 4, 2021 11:42:05 AM CET    November 4, 2021 11:42:07 AM CET    Success

The final steps is then to re-run the update server command:

[root@dbi-oda-x8 bin]# odacli update-server -v 19.12.0.0.0
{
  "jobId" : "946922b3-aefa-4c75-b2e1-019e9d404118",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "November 04, 2021 11:42:57 AM CET",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "November 04, 2021 11:42:57 AM CET"
}

After few minutes the patching of the server is finally successful 🙂 🙂

[root@dbi-oda-x8 ~]# odacli describe-job -i 946922b3-aefa-4c75-b2e1-019e9d404118

Job details
----------------------------------------------------------------
                     ID:  946922b3-aefa-4c75-b2e1-019e9d404118
            Description:  Server Patching
                 Status:  Success
                Created:  November 4, 2021 11:42:57 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating repositories using yum          November 4, 2021 11:43:11 AM CET    November 4, 2021 11:43:13 AM CET    Success
Updating YumPluginVersionLock rpm        November 4, 2021 11:43:13 AM CET    November 4, 2021 11:43:13 AM CET    Success
Applying OS Patches                      November 4, 2021 11:43:13 AM CET    November 4, 2021 11:43:14 AM CET    Success
Creating repositories using yum          November 4, 2021 11:43:15 AM CET    November 4, 2021 11:43:15 AM CET    Success
Applying HMP Patches                     November 4, 2021 11:43:15 AM CET    November 4, 2021 11:43:16 AM CET    Success
Patch location validation                November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:16 AM CET    Success
oda-hw-mgmt upgrade                      November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:16 AM CET    Success
OSS Patching                             November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:16 AM CET    Success
Applying Firmware Disk Patches           November 4, 2021 11:43:16 AM CET    November 4, 2021 11:43:20 AM CET    Success
Applying Firmware Controller Patches     November 4, 2021 11:43:20 AM CET    November 4, 2021 11:43:24 AM CET    Success
Checking Ilom patch Version              November 4, 2021 11:43:24 AM CET    November 4, 2021 11:43:24 AM CET    Success
Patch location validation                November 4, 2021 11:43:24 AM CET    November 4, 2021 11:43:24 AM CET    Success
Save password in Wallet                  November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
Apply Ilom patch                         November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
Copying Flash Bios to Temp location      November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
Starting the clusterware                 November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:25 AM CET    Success
registering image                        November 4, 2021 11:43:25 AM CET    November 4, 2021 11:43:26 AM CET    Success
registering working copy                 November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
registering image                        November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Creating GI home directories             November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Extract GI clone                         November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Provisioning Software Only GI with RHP   November 4, 2021 11:43:26 AM CET    November 4, 2021 11:43:26 AM CET    Success
Patch GI with RHP                        November 4, 2021 11:43:26 AM CET    November 4, 2021 11:50:05 AM CET    Success
Updating GIHome version                  November 4, 2021 11:50:06 AM CET    November 4, 2021 11:50:08 AM CET    Success
Update System version                    November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:23 AM CET    Success
Cleanup JRE Home                         November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:23 AM CET    Success
Add SYSNAME in Env                       November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:23 AM CET    Success
Setting ACL for disk groups              November 4, 2021 11:50:23 AM CET    November 4, 2021 11:50:27 AM CET    Success
preRebootNode Actions                    November 4, 2021 11:52:14 AM CET    November 4, 2021 11:55:06 AM CET    Success
Reboot Ilom                              November 4, 2021 11:55:06 AM CET    November 4, 2021 11:55:06 AM CET    Success

Here we go our ODA is now up-to-date in version 19.12, we can continue with:

• cleaning the repo
• manually cleaning the former GI Home
• Creating a Database Home Store
• …
• … 😀

Enjoy! 😎