Patching ODA HA from 19.8 to 19.10

In this blog I am describing the steps to patch an ODA HA from the version 19.8 to version 19.10. The patching was done with a customer. Thanks to Pierre. The name of the servers were changed
The ODA is an X8-HA with following nodes
-server-aa
-server-ab

Before starting the patches, be sure that you have enough space on / /u01 and /opt filesystems. Below the state of our filesystems

[root@server-aa Downloads]# df -h / /opt /u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G   19G  9.7G  66% /
/dev/mapper/VolGroupSys-LogVolOpt    99G   54G   40G  58% /opt
/dev/mapper/VolGroupSys-LogVolU01    99G   34G   60G  36% /u01

[root@server-ab oracle]# df -h / /opt /u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G  7.6G   21G  28% /
/dev/mapper/VolGroupSys-LogVolOpt    99G   54G   40G  58% /opt
/dev/mapper/VolGroupSys-LogVolU01    99G   33G   62G  35% /u01

We also take snapshots with odabr on both nodes. See my previous blog

For the patch of the server you will need the following patchs ( to be downloaded from support.oracle.com)
-p32351355_1910000_Linux-x86-64_1of2.zip
-p32351355_1910000_Linux-x86-64_2of2.zip

The steps are the following. All steps can be found in the documentaion

1- Unpack the 2 files and then copy them to the first node

2- Update the repository with the 2 files

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-repository -f /opt/oracle/Downloads/oda-sm-19.10.0.0.0-210222.4-server1of2.zip

Then verify that the status of the job returns success

[root@server-aa Downloads]# odacli describe-job -i "36206afa-438d-4556-85a7-1a92f5feb68d"

Do the same for the second file

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-repository -f /opt/oracle/Downloads/oda-sm-19.10.0.0.0-210222.4-server2of2.zip

Check the status

[root@server-aa Downloads]# odacli describe-job -i "d6320bf3-58b5-4b39-b557-27074caeb785"

After the update of the repository you can remove the 2 files to save space

3- Update the DCS admin

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.10.0.0.0

Verify the status returns success

[root@server-aa Downloads]# odacli describe-job -i "cd29d048-e02e-4499-8484-6e8e5079fc3b"

4- Update the DCS components

The command above may take some times as specified in the documentation. So be patient and wait
This command updates the Zookeeper, installs MySQL, migrates metadata from Derby to MySQL and updates other DCS components such as the DCS Agent, DCS CLI, and DCS Controller on Oracle Database Appliance.

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.10.0.0.0

5- Update the DCS agent

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.10.0.0.0

Check the status

[root@server-aa Downloads]# odacli describe-job -i "d4d3c989-0c3b-4cf0-9ba3-08a51a3306cf"

6- Create a precheck-report

Before updating the server, we run the prechecks

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.10.0.0.0

We present below the ouputs of our report

[root@server-aa Downloads]# odacli describe-prepatchreport -i 7d2b9a84-effe-4420-9732-8edffd53bb04

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  7d2b9a84-effe-4420-9732-8edffd53bb04
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  FAILED
                Created:  May 3, 2021 11:15:58 AM CEST
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
server-aa

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.10.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.10.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.10.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.10.0.0.0.
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 32218454 not applied on GI
                                          home /u01/app/19.0.0.0/oracle on node
                                          server-aa
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/oracle on
                                          node server-aa
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/oracle on node server-aa
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Verify the vm.min_free_kbytes   Failed    AHF-4819: The vm.min_free_kbytes
configuration                             configuration is not set as
                                          recommended

Node Name
---------------
server-ab

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.10.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.10.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.10.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate GI patch metadata      Success   Validated patching tag: 19.10.0.0.0.
Is patch location available     Success   Patch location is available.
Patch location validation       Success   Successfully validated location
Patch verification              Success   Patches 32218454 not applied on GI
                                          home /u01/app/19.0.0.0/oracle on node
                                          server-ab
Validate Opatch update          Success   Successfully updated the opatch in
                                          GiHome /u01/app/19.0.0.0/oracle on
                                          node server-ab
Patch conflict check            Success   No patch conflicts found on GiHome
                                          /u01/app/19.0.0.0/oracle on node server-ab
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Verify the vm.min_free_kbytes   Failed    AHF-4819: The vm.min_free_kbytes
configuration                             configuration is not set as
                                          recommended

In the results we can see that there are some errors due to orachk. Based on following document https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/19.10/cmtrn/issues-with-oda-odacli.html#GUID-F2B10F21-3D1E-4328-8E9B-D75AD38D26A1 , we decide to ignore the errors and then continue with updating the server

7- Apply the server update

As we ignore the error we use the flag sko for the patching

[root@server-aa log]# /opt/oracle/dcs/bin/odacli update-server -v 19.10.0.0.0 -sko

A few moment after the status return success

[root@server-aa ~]# odacli describe-job -i 56aa8af1-d59e-4e78-b3e5-8fc17599f73f

Job details
----------------------------------------------------------------
                     ID:  56aa8af1-d59e-4e78-b3e5-8fc17599f73f
            Description:  Server Patching
                 Status:  Success
                Created:  May 3, 2021 11:52:36 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                May 3, 2021 11:52:50 AM CEST        May 3, 2021 11:52:50 AM CEST        Success
Patch location validation                May 3, 2021 11:52:50 AM CEST        May 3, 2021 11:52:50 AM CEST        Success
dcs-controller upgrade                   May 3, 2021 11:52:50 AM CEST        May 3, 2021 11:52:50 AM CEST        Success
dcs-controller upgrade                   May 3, 2021 11:52:50 AM CEST        May 3, 2021 11:52:50 AM CEST        Success
Creating repositories using yum          May 3, 2021 11:52:53 AM CEST        May 3, 2021 11:52:55 AM CEST        Success
Updating YumPluginVersionLock rpm        May 3, 2021 11:52:55 AM CEST        May 3, 2021 11:52:55 AM CEST        Success
Applying OS Patches                      May 3, 2021 11:52:55 AM CEST        May 3, 2021 12:01:30 PM CEST        Success
Creating repositories using yum          May 3, 2021 12:01:42 PM CEST        May 3, 2021 12:01:42 PM CEST        Success
Applying HMP Patches                     May 3, 2021 12:01:42 PM CEST        May 3, 2021 12:01:43 PM CEST        Success
Client root Set up                       May 3, 2021 12:01:43 PM CEST        May 3, 2021 12:01:43 PM CEST        Success
Client oracle Set up                     May 3, 2021 12:01:43 PM CEST        May 3, 2021 12:01:43 PM CEST        Success
Patch location validation                May 3, 2021 12:01:43 PM CEST        May 3, 2021 12:01:43 PM CEST        Success
Patch location validation                May 3, 2021 12:01:43 PM CEST        May 3, 2021 12:01:43 PM CEST        Success
oda-hw-mgmt upgrade                      May 3, 2021 12:01:44 PM CEST        May 3, 2021 12:02:14 PM CEST        Success
oda-hw-mgmt upgrade                      May 3, 2021 12:02:14 PM CEST        May 3, 2021 12:02:44 PM CEST        Success
OSS Patching                             May 3, 2021 12:02:44 PM CEST        May 3, 2021 12:02:45 PM CEST        Success
Applying Firmware Disk Patches           May 3, 2021 12:02:48 PM CEST        May 3, 2021 12:02:51 PM CEST        Success
Applying Firmware Controller Patches     May 3, 2021 12:02:53 PM CEST        May 3, 2021 12:02:56 PM CEST        Success
Checking Ilom patch Version              May 3, 2021 12:02:56 PM CEST        May 3, 2021 12:02:56 PM CEST        Success
Checking Ilom patch Version              May 3, 2021 12:02:56 PM CEST        May 3, 2021 12:02:56 PM CEST        Success
Patch location validation                May 3, 2021 12:02:56 PM CEST        May 3, 2021 12:02:56 PM CEST        Success
Patch location validation                May 3, 2021 12:02:56 PM CEST        May 3, 2021 12:02:56 PM CEST        Success
Save password in Wallet                  May 3, 2021 12:02:56 PM CEST        May 3, 2021 12:02:58 PM CEST        Success
Apply Ilom patch                         May 3, 2021 12:02:58 PM CEST        May 3, 2021 12:11:50 PM CEST        Success
Apply Ilom patch                         May 3, 2021 12:11:50 PM CEST        May 3, 2021 12:20:39 PM CEST        Success
Copying Flash Bios to Temp location      May 3, 2021 12:20:39 PM CEST        May 3, 2021 12:20:39 PM CEST        Success
Copying Flash Bios to Temp location      May 3, 2021 12:20:39 PM CEST        May 3, 2021 12:20:39 PM CEST        Success
Starting the clusterware                 May 3, 2021 12:22:20 PM CEST        May 3, 2021 12:23:39 PM CEST        Success
create ASM priv network                  May 3, 2021 12:23:39 PM CEST        May 3, 2021 12:23:44 PM CEST        Success
create ASM priv network                  May 3, 2021 12:23:44 PM CEST        May 3, 2021 12:23:48 PM CEST        Success
configure ASM priv network               May 3, 2021 12:23:48 PM CEST        May 3, 2021 12:23:51 PM CEST        Success
configure ASM priv network               May 3, 2021 12:23:51 PM CEST        May 3, 2021 12:23:51 PM CEST        Success
Starting the clusterware                 May 3, 2021 12:23:53 PM CEST        May 3, 2021 12:23:53 PM CEST        Success
clusterware patch verification           May 3, 2021 12:23:53 PM CEST        May 3, 2021 12:23:56 PM CEST        Success
clusterware patch verification           May 3, 2021 12:23:53 PM CEST        May 3, 2021 12:23:56 PM CEST        Success
Patch location validation                May 3, 2021 12:23:56 PM CEST        May 3, 2021 12:23:56 PM CEST        Success
Patch location validation                May 3, 2021 12:23:56 PM CEST        May 3, 2021 12:23:56 PM CEST        Success
Opatch update                            May 3, 2021 12:24:50 PM CEST        May 3, 2021 12:24:53 PM CEST        Success
Opatch update                            May 3, 2021 12:24:50 PM CEST        May 3, 2021 12:24:53 PM CEST        Success
Patch conflict check                     May 3, 2021 12:24:53 PM CEST        May 3, 2021 12:26:03 PM CEST        Success
Patch conflict check                     May 3, 2021 12:26:03 PM CEST        May 3, 2021 12:27:11 PM CEST        Success
clusterware upgrade                      May 3, 2021 12:27:41 PM CEST        May 3, 2021 1:18:19 PM CEST         Success
clusterware upgrade                      May 3, 2021 1:18:19 PM CEST         May 3, 2021 2:03:14 PM CEST         Success
Updating GiHome version                  May 3, 2021 2:03:14 PM CEST         May 3, 2021 2:03:17 PM CEST         Success
Updating GiHome version                  May 3, 2021 2:03:14 PM CEST         May 3, 2021 2:03:18 PM CEST         Success
Starting the clusterware                 May 3, 2021 2:03:58 PM CEST         May 3, 2021 2:03:58 PM CEST         Success
remove network public interface          May 3, 2021 2:04:05 PM CEST         May 3, 2021 2:04:12 PM CEST         Success
create bridge network                    May 3, 2021 2:04:12 PM CEST         May 3, 2021 2:04:17 PM CEST         Success
create bridge network                    May 3, 2021 2:04:17 PM CEST         May 3, 2021 2:04:22 PM CEST         Success
modify network public interface          May 3, 2021 2:04:22 PM CEST         May 3, 2021 2:04:31 PM CEST         Success
Update System version                    May 3, 2021 2:04:31 PM CEST         May 3, 2021 2:04:31 PM CEST         Success
Update System version                    May 3, 2021 2:04:31 PM CEST         May 3, 2021 2:04:31 PM CEST         Success
Cleanup JRE Home                         May 3, 2021 2:04:31 PM CEST         May 3, 2021 2:04:31 PM CEST         Success
Add SYSNAME in Env                       May 3, 2021 2:04:31 PM CEST         May 3, 2021 2:04:31 PM CEST         Success
Setting ACL for disk groups              May 3, 2021 2:04:31 PM CEST         May 3, 2021 2:04:34 PM CEST         Success
preRebootNode Actions                    May 3, 2021 2:04:34 PM CEST         May 3, 2021 2:05:15 PM CEST         Success
preRebootNode Actions                    May 3, 2021 2:05:15 PM CEST         May 3, 2021 2:06:01 PM CEST         Success
Reboot Ilom                              May 3, 2021 2:06:01 PM CEST         May 3, 2021 2:06:01 PM CEST         Success
Reboot Ilom                              May 3, 2021 2:06:01 PM CEST         May 3, 2021 2:06:01 PM CEST         Success

We can confirm this by running the following command

[root@server-aa ~]# odacli describe-component
System Version
---------------
19.10.0.0.0

System node Name
---------------
server-aa

Local System Version
---------------
19.10.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.10.0.0.0           up-to-date

GI                                        19.10.0.0.210119      up-to-date

DB                                        12.2.0.1.201020       12.2.0.1.210119

DCSAGENT                                  19.10.0.0.0           up-to-date

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    up-to-date

BIOS                                      52030400              up-to-date

FIRMWARECONTROLLER                        16.00.08.00           up-to-date

FIRMWAREEXPANDER                          0310                  up-to-date

FIRMWAREDISK {
[ c1d0,c1d1 ]                             1120                  1132
[ c0d0,c0d1,c0d2,c0d3,c0d4,c0d5,c0d6,     A959                  up-to-date
c0d7,c0d8,c0d9,c0d10,c0d11,c2d0,c2d1,
c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,c2d9,
c2d10,c2d11 ]
}

HMP                                       2.4.7.0.1             up-to-date

System node Name
---------------
server-ab

Local System Version
---------------
19.10.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.10.0.0.0           up-to-date

GI                                        19.10.0.0.210119      up-to-date

DB                                        12.2.0.1.201020       12.2.0.1.210119

DCSAGENT                                  19.10.0.0.0           up-to-date

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    up-to-date

BIOS                                      52030400              up-to-date

FIRMWARECONTROLLER                        16.00.08.00           up-to-date

FIRMWAREEXPANDER                          0310                  up-to-date

FIRMWAREDISK {
[ c1d0,c1d1 ]                             1120                  1132
[ c0d0,c0d1,c0d2,c0d3,c0d4,c0d5,c0d6,     A959                  up-to-date
c0d7,c0d8,c0d9,c0d10,c0d11,c2d0,c2d1,
c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,c2d8,c2d9,
c2d10,c2d11 ]
}

HMP                                       2.4.7.0.1             up-to-date

8- Update the storage components

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-storage -v 19.10.0.0.0 --rolling

Check the status

[root@server-aa ~]# odacli describe-job -i "9d4893e5-3ebd-4b38-a746-32262bbdcb2e"

9- Update the existing dbhomes

List your dbhomes

[root@server-aa ~]# odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
379c50b9-9fd3-4049-8ceb-8440d8c2285d     OraDB12201_home2     12.2.0.1.201020                          /u01/app/oracle/product/12.2.0.1/dbhome_2     CONFIGURED

Run the prepatch report

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli create-prepatchreport --dbhome --dbhomeid 379c50b9-9fd3-4049-8ceb-8440d8c2285d -v 19.10.0.0.0

Check the status

[root@server-aa ~]# odacli describe-prepatchreport -i d2d676bc-f802-4f2d-a748-b43c601407fd

And then update the corresponding home

[root@server-aa Downloads]# /opt/oracle/dcs/bin/odacli update-dbhome --dbhomeid 379c50b9-9fd3-4049-8ceb-8440d8c2285d -v 19.10.0.0.0

Check the status

[root@server-aa ~]# odacli describe-job -i "ae2d7014-eb9a-468e-8b24-85ec6a20f039"

Now that the patch was successful, we can delete the snapshots we took with odabr

[root@server-aa ~]# /opt/odabr/odabr delsnap
[root@server-ab ~]# /opt/odabr/odabr delsnap

10- Update the repository with the rdbms clone 19.10

The last step is to update the repository with the rdbms clone 19.10. For this download the patch p30403662_199000_Linux-x86-64.zip. And then unpack the file and run the command

[root@server-aa Downloads]#
/opt/oracle/dcs/bin/odacli update-repository -f /opt/oracle/Downloads/odacli-dcs-19.10.0.0.0-210208-DB-19.10.0.0.zip

And check that the status returns success

[root@server-aa Downloads]# odacli describe-job -i "4368d49e-7e00-4c12-8ddb-80cb90bfab82"