By Mouhamadou Diaw

Patching an ODA is always exciting. Preparing a patch of an ODA HA may seem more complicated that an single ODA. But it’s the same processus. All patching operations should be run on one node.

In this document I am describing the patch of a ODA HA from 19.14 to 19.16 with a customer. Great thanks to Pierre.

The documentation is well done. Just read and followw the steps

We downloaded following patches

Oracle Database Appliance Server Patch for the ODACLI/DCS stack : p34376263_1916000_Linux-x86-64.zip
Oracle Grid Infrastructure clone files : p30403673_1916000_Linux-x86-64.zip
Oracle Database clone files : p30403662_1916000_Linux-x86-64.zip

Once unpacked we will have following files

oda-sm-19.16.0.0.0-220809-server.zip
odacli-dcs-19.16.0.0.0-220717-GI-19.16.0.0.zip
odacli-dcs-19.16.0.0.0-220717-DB-19.16.0.0.zip

These files will be used to update the repository. They should be copied in the ODA server.

Note that all operations will be done only on a node. If some tasks need to be done on both nodes, we will specify

On one node of the ODA, connect as root and  update the repository with the patch for the ODACLI/DCS stack

[root@oda-deva]#  /opt/oracle/dcs/bin/odacli update-repository -f /opt/Downloads/oda-sm-19.16.0.0.0-220809-server.zip

Validate that the job was successful

[2022-10-12 11:43:04 root@oda-deva]# odacli describe-job -i "7a58f486-6ab1-4b4f-b9a6-a38458535d2c"

Job details
----------------------------------------------------------------
                     ID:  7a58f486-6ab1-4b4f-b9a6-a38458535d2c
            Description:  Repository Update
                 Status:  Success
                Created:  October 12, 2022 11:42:25 AM CEST
                Message:  /opt/Downloads/oda-sm-19.16.0.0.0-220809-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     October 12, 2022 11:42:32 AM CEST   October 12, 2022 11:42:32 AM CEST   Success
Setting up ssh equivalance               October 12, 2022 11:42:32 AM CEST   October 12, 2022 11:42:32 AM CEST   Success
Copy BundleFile                          October 12, 2022 11:42:33 AM CEST   October 12, 2022 11:42:43 AM CEST   Success
Validating CopiedFile                    October 12, 2022 11:42:43 AM CEST   October 12, 2022 11:42:49 AM CEST   Success
Unzip bundle                             October 12, 2022 11:42:49 AM CEST   October 12, 2022 11:43:12 AM CEST   Success
Unzip bundle                             October 12, 2022 11:43:18 AM CEST   October 12, 2022 11:43:45 AM CEST   Success
Removing ssh keys                        October 12, 2022 11:43:45 AM CEST   October 12, 2022 11:43:46 AM CEST   Success

Then update the DCS admin with user root

# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.16.0.0.0

Validate the status of the job

[2022-10-12 11:45:57 root@oda-deva]# odacli describe-job -i "8d792424-e5c0-4ec5-8133-606a6d3632cb"

Job details
----------------------------------------------------------------
                     ID:  8d792424-e5c0-4ec5-8133-606a6d3632cb
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  October 12, 2022 11:45:57 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                October 12, 2022 11:45:57 AM CEST   October 12, 2022 11:45:57 AM CEST   Success
Patch location validation                October 12, 2022 11:45:57 AM CEST   October 12, 2022 11:45:57 AM CEST   Success
dcs-admin upgrade                        October 12, 2022 11:45:58 AM CEST   October 12, 2022 11:46:04 AM CEST   Success
dcs-admin upgrade                        October 12, 2022 11:46:04 AM CEST   October 12, 2022 11:46:10 AM CEST   Success

After update the DCS components with root

[2022-10-12 11:46:49 root@oda-deva]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.16.0.0.0
{
  "jobId" : "2bc1aebd-50a7-41fe-b66e-b08ed0bc83d1",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  ",
  "reports" : null,
  "createTimestamp" : "October 12, 2022 11:47:03 AM CEST",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "October 12, 2022 11:50:34 AM CEST"
}

The next step is to update the DCS agent with the user root

# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.16.0.0.0

We still have to validate that the job was successfull

[2022-10-12 11:56:25 root@oda-deva]# odacli describe-job -i "350ebfca-0ed9-48da-8ed7-ba37dba1df44"

Job details
----------------------------------------------------------------
                     ID:  350ebfca-0ed9-48da-8ed7-ba37dba1df44
            Description:  DcsAgent patching
                 Status:  Success
                Created:  October 12, 2022 11:51:53 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DCS Admin                           October 12, 2022 11:51:59 AM CEST   October 12, 2022 11:51:59 AM CEST   Success
Stop DCS Admin                           October 12, 2022 11:51:59 AM CEST   October 12, 2022 11:51:59 AM CEST   Success
Generate mTLS certificates               October 12, 2022 11:51:59 AM CEST   October 12, 2022 11:52:00 AM CEST   Success
Generate mTLS certificates               October 12, 2022 11:51:59 AM CEST   October 12, 2022 11:52:01 AM CEST   Success
Exporting Public Keys                    October 12, 2022 11:52:01 AM CEST   October 12, 2022 11:52:02 AM CEST   Success
Exporting Public Keys                    October 12, 2022 11:52:01 AM CEST   October 12, 2022 11:52:02 AM CEST   Success
Copy Public Key to Remote node           October 12, 2022 11:52:02 AM CEST   October 12, 2022 11:52:03 AM CEST   Success
Copy Public Key to Remote node           October 12, 2022 11:52:03 AM CEST   October 12, 2022 11:52:04 AM CEST   Success
Creating Trust Store                     October 12, 2022 11:52:04 AM CEST   October 12, 2022 11:52:06 AM CEST   Success
Creating Trust Store                     October 12, 2022 11:52:04 AM CEST   October 12, 2022 11:52:06 AM CEST   Success
Update config files                      October 12, 2022 11:52:06 AM CEST   October 12, 2022 11:52:06 AM CEST   Success
Update config files                      October 12, 2022 11:52:06 AM CEST   October 12, 2022 11:52:06 AM CEST   Success
Restart DCS Admin                        October 12, 2022 11:52:06 AM CEST   October 12, 2022 11:52:27 AM CEST   Success
Restart DCS Admin                        October 12, 2022 11:52:06 AM CEST   October 12, 2022 11:52:27 AM CEST   Success
Dcs-agent upgrade  to version 19.16.0.0.0 October 12, 2022 11:52:27 AM CEST   October 12, 2022 11:54:27 AM CEST   Success
Dcs-agent upgrade  to version 19.16.0.0.0 October 12, 2022 11:54:27 AM CEST   October 12, 2022 11:56:27 AM CEST   Success
Update System version                    October 12, 2022 11:56:28 AM CEST   October 12, 2022 11:56:28 AM CEST   Success
Update System version                    October 12, 2022 11:56:28 AM CEST   October 12, 2022 11:56:28 AM CEST   Success

Now it’s time to update the repository with the GI clone we downloaded

# /opt/oracle/dcs/bin/odacli update-repository -f /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-GI-19.16.0.0.zip

And to validate the status of the job

[2022-10-12 12:03:17 root@oda-deva]# odacli describe-job -i "921447ac-be4c-470d-a683-bb4d5bc287ec"

Job details
----------------------------------------------------------------
                     ID:  921447ac-be4c-470d-a683-bb4d5bc287ec
            Description:  Repository Update
                 Status:  Success
                Created:  October 12, 2022 12:00:37 PM CEST
                Message:  /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-GI-19.16.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     October 12, 2022 12:00:50 PM CEST   October 12, 2022 12:00:50 PM CEST   Success
Setting up ssh equivalance               October 12, 2022 12:00:50 PM CEST   October 12, 2022 12:00:53 PM CEST   Success
Copy BundleFile                          October 12, 2022 12:00:53 PM CEST   October 12, 2022 12:01:16 PM CEST   Success
Validating CopiedFile                    October 12, 2022 12:01:16 PM CEST   October 12, 2022 12:01:43 PM CEST   Success
Unzip bundle                             October 12, 2022 12:01:49 PM CEST   October 12, 2022 12:02:37 PM CEST   Success
Unzip bundle                             October 12, 2022 12:02:37 PM CEST   October 12, 2022 12:03:25 PM CEST   Success
Delete PatchBundles                      October 12, 2022 12:03:25 PM CEST   October 12, 2022 12:03:26 PM CEST   Success

Update the repository with the DB clone

# /opt/oracle/dcs/bin/odacli update-repository -f /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-DB-19.16.0.0.zip
[2022-10-12 12:05:52 root@oda-deva]# odacli describe-job -i "ad8eed4c-7c2e-4e76-b984-e0e19467d909"

Job details
----------------------------------------------------------------
                     ID:  ad8eed4c-7c2e-4e76-b984-e0e19467d909
            Description:  Repository Update
                 Status:  Success
                Created:  October 12, 2022 12:04:03 PM CEST
                Message:  /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-DB-19.16.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     October 12, 2022 12:04:14 PM CEST   October 12, 2022 12:04:14 PM CEST   Success
Setting up ssh equivalance               October 12, 2022 12:04:14 PM CEST   October 12, 2022 12:04:15 PM CEST   Success
Copy BundleFile                          October 12, 2022 12:04:15 PM CEST   October 12, 2022 12:04:31 PM CEST   Success
Validating CopiedFile                    October 12, 2022 12:04:31 PM CEST   October 12, 2022 12:04:41 PM CEST   Success
Unzip bundle                             October 12, 2022 12:04:41 PM CEST   October 12, 2022 12:05:19 PM CEST   Success
Unzip bundle                             October 12, 2022 12:05:19 PM CEST   October 12, 2022 12:05:57 PM CEST   Success
Delete PatchBundles                      October 12, 2022 12:05:57 PM CEST   October 12, 2022 12:05:58 PM CEST   Success

At this moment, we can check the status of the components

# odacli describe-component
	[2022-10-12 12:06:12 root@oda-deva]# odacli describe-component
	System Version
	---------------
	19.16.0.0.0

	System node Name
	---------------
	oda-deva

	Local System Version
	---------------
	19.16.0.0.0

	Component                                Installed Version    Available Version
	---------------------------------------- -------------------- --------------------
	OAK
											  19.14.0.0.0           19.16.0.0.0
	GI
											  19.14.0.0.220118      19.16.0.0.220719
	DB[ OFCCMPRE,T24DEV01,C1IDPDEV,C1IDPPRE,
	C1DEVOPS ]                                19.14.0.0.220118      19.16.0.0.220719
	DCSCONTROLLER
											  19.16.0.0.0           up-to-date
	DCSCLI
											  19.16.0.0.0           up-to-date
	DCSAGENT
											  19.16.0.0.0           up-to-date
	DCSADMIN
											  19.16.0.0.0           up-to-date
	OS
											  7.9                   up-to-date
	ILOM
											  5.0.1.21.a.r138015    5.1.0.20.r145377
	BIOS
											  41080800              41110100
	LOCAL CONTROLLER FIRMWARE {
	[c3,c4]
											  20.08.01.18           214.2.271.9
	[c5]
											  80000690              up-to-date
	}
	SHARED CONTROLLER FIRMWARE
											  16.00.08.00           up-to-date
	FIRMWAREEXPANDER
											  0310                  up-to-date
	LOCAL DISK FIRMWARE
											  N2010121              up-to-date
	SHARED DISK FIRMWARE {
	[c0d0,c0d1]
											  N2010121              up-to-date
	[c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
	c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
	c1d14,c1d15,c1d16,c1d17,c1d18,c1d19,
	c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,
	c2d8,c2d9,c2d10,c2d11,c2d12,c2d13,c2d14,
	c2d15,c2d16,c2d17,c2d18,c2d19]            A170                  up-to-date
	[c1d20,c1d21,c1d22,c1d23,c2d20,c2d21,
	c2d22,c2d23]                              A959                  A902
	}
	HMP
											  2.4.8.0.600           2.4.8.9.601
System node Name
	---------------
	oda-devb

	Local System Version
	---------------
	19.16.0.0.0

	Component                                Installed Version    Available Version
	---------------------------------------- -------------------- --------------------
	OAK
											  19.14.0.0.0           19.16.0.0.0
	GI
											  19.14.0.0.220118      19.16.0.0.220719
	DB[ OFCCMPRE,T24DEV01,C1IDPDEV,C1IDPPRE,
	C1DEVOPS ]                                19.14.0.0.220118      19.16.0.0.220719
	DCSCONTROLLER
											  19.16.0.0.0           up-to-date
	DCSCLI
											  19.16.0.0.0           up-to-date
	DCSAGENT
											  19.16.0.0.0           up-to-date
	DCSADMIN
											  19.16.0.0.0           up-to-date
	OS
											  7.9                   up-to-date
	ILOM
											  5.0.1.21.a.r138015    5.1.0.20.r145377
	BIOS
											  41080800              41110100
	LOCAL CONTROLLER FIRMWARE {
	[c3]
											  80000690              up-to-date
	[c4,c5]
											  20.08.01.18           214.2.271.9
	}
	SHARED CONTROLLER FIRMWARE
											  16.00.08.00           up-to-date
	FIRMWAREEXPANDER
											  0310                  up-to-date
	LOCAL DISK FIRMWARE
											  N2010121              up-to-date
	SHARED DISK FIRMWARE {
	[c0d0,c0d1]
											  N2010121              up-to-date
	[c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
	c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
	c1d14,c1d15,c1d16,c1d17,c1d18,c1d19,
	c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,
	c2d8,c2d9,c2d10,c2d11,c2d12,c2d13,c2d14,
	c2d15,c2d16,c2d17,c2d18,c2d19]            A170                  up-to-date
	[c1d20,c1d21,c1d22,c1d23,c2d20,c2d21,
	c2d22,c2d23]                              A959                  A902
	}
	HMP
											  2.4.8.0.600           2.4.8.9.601

Before updating the server, we do generate the prepatchreport

[2022-10-12 12:07:39 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.16.0.0.0

Job details
----------------------------------------------------------------
                     ID:  a0087a74-994f-4097-b32c-6c024d9a05d7
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  Created
                Created:  October 12, 2022 12:08:14 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i a0087a74-994f-4097-b32c-6c024d9a05d7' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

This report should return any error.

[2022-10-12 12:27:25 root@oda-deva]# odacli describe-prepatchreport -i a0087a74-994f-4097-b32c-6c024d9a05d7

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  a0087a74-994f-4097-b32c-6c024d9a05d7
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  FAILED
                Created:  October 12, 2022 12:08:14 PM CEST
                 Result:  One or more pre-checks failed for [GI]

Node Name
---------------
oda-deva

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Failed    Internal error encountered: Lock on
                                          Central Inventory detected. This may
                                          result in a failure patching the GI.
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

Node Name
---------------
oda-devb

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Failed    Internal error encountered: Lock on
                                          Central Inventory detected. This may
                                          result in a failure patching the GI.
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

As we can see above we got errors about lock on Central Inventory. We just follow the steps from this Oracle documentation on both nodes

ODA Upgrade PrepatchReport Fails Due To OraInventory Lock Internal Error (Doc ID 2659378.1)

[2022-10-12 13:33:00 root@oda-deva]# cd /u01/app/oraInventory/
[2022-10-12 13:33:15 root@oda-deva]# ls -l
total 24
drwxrwx--- 6 oracle oinstall 4096 Oct 12 12:15 backup
drwxrwx--- 2 oracle oinstall 4096 Oct 12 12:15 ContentsXML
drwxrwx--- 2 oracle oinstall 4096 Oct 12 12:15 locks
drwxrwx--- 7 oracle oinstall 4096 Oct 12 12:14 logs
-rw-rw---- 1 oracle oinstall   56 Apr 20 16:16 oraInst.loc
-rwxrwx--- 1 oracle oinstall 1617 Apr 20 16:16 orainstRoot.sh

[2022-10-12 13:33:18 root@oda-deva]# mv locks locks.221012

And we rerun the prepatch report

[2022-10-12 13:33:40 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.16.0.0.0


Job details
----------------------------------------------------------------
                     ID:  2d2a2042-31e0-4750-af90-605f6b3bbb20
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  Created
                Created:  October 12, 2022 1:35:34 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i 2d2a2042-31e0-4750-af90-605f6b3bbb20' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

And the prepatch report was successful

[2022-10-12 13:45:44 root@oda-deva]# odacli describe-prepatchreport -i 2d2a2042-31e0-4750-af90-605f6b3bbb20

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  2d2a2042-31e0-4750-af90-605f6b3bbb20
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  SUCCESS
                Created:  October 12, 2022 1:35:34 PM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
oda-deva

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

Node Name
---------------
oda-devb

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

As all checks are fine, we can now run the update server command

[2022-10-12 13:46:19 root@oda-deva]# /opt/oracle/dcs/bin/odacli update-server -v 19.16.0.0.0
{
  "jobId" : "a03fce0a-413c-458f-a051-0537109c66f1",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "October 12, 2022 13:47:11 PM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "October 12, 2022 13:47:11 PM CEST",
  "jobType" : null
}

Few times after, the update server was successful

[2022-10-12 15:08:10 root@oda-deva]# odacli describe-job -i a03fce0a-413c-458f-a051-0537109c66f1

Job details
----------------------------------------------------------------
                     ID:  a03fce0a-413c-458f-a051-0537109c66f1
            Description:  Server Patching
                 Status:  Success
                Created:  October 12, 2022 1:47:11 PM CEST
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              October 12, 2022 1:47:37 PM CEST    October 12, 2022 1:47:37 PM CEST    Success
Validating GI user metadata              October 12, 2022 1:47:37 PM CEST    October 12, 2022 1:47:37 PM CEST    Success
Validate ILOM server reachable           October 12, 2022 1:47:37 PM CEST    October 12, 2022 1:47:37 PM CEST    Success
Validate ILOM server reachable           October 12, 2022 1:47:37 PM CEST    October 12, 2022 1:47:37 PM CEST    Success
Validate DCS Admin mTLS setup            October 12, 2022 1:47:37 PM CEST    October 12, 2022 1:47:37 PM CEST    Success
Validate DCS Admin mTLS setup            October 12, 2022 1:47:37 PM CEST    October 12, 2022 1:47:37 PM CEST    Success
Configure export clones resource         October 12, 2022 1:47:40 PM CEST    October 12, 2022 1:47:40 PM CEST    Success
Creating repositories using yum          October 12, 2022 1:47:43 PM CEST    October 12, 2022 1:47:46 PM CEST    Success
Updating YumPluginVersionLock rpm        October 12, 2022 1:47:46 PM CEST    October 12, 2022 1:47:46 PM CEST    Success
Applying OS Patches                      October 12, 2022 1:47:46 PM CEST    October 12, 2022 1:59:44 PM CEST    Success
Creating repositories using yum          October 12, 2022 1:59:45 PM CEST    October 12, 2022 1:59:46 PM CEST    Success
Applying HMP Patches                     October 12, 2022 1:59:46 PM CEST    October 12, 2022 2:00:05 PM CEST    Success
Patch location validation                October 12, 2022 2:00:06 PM CEST    October 12, 2022 2:00:06 PM CEST    Success
Patch location validation                October 12, 2022 2:00:06 PM CEST    October 12, 2022 2:00:06 PM CEST    Success
Oda-hw-mgmt upgrade                      October 12, 2022 2:00:07 PM CEST    October 12, 2022 2:00:40 PM CEST    Success
Oda-hw-mgmt upgrade                      October 12, 2022 2:00:40 PM CEST    October 12, 2022 2:01:14 PM CEST    Success
OSS Patching                             October 12, 2022 2:01:14 PM CEST    October 12, 2022 2:01:14 PM CEST    Success
Applying Firmware Disk Patches           October 12, 2022 2:01:18 PM CEST    October 12, 2022 2:01:23 PM CEST    Success
Applying Firmware Controller Patches     October 12, 2022 2:03:17 PM CEST    October 12, 2022 2:05:14 PM CEST    Success
Checking Ilom patch Version              October 12, 2022 2:05:14 PM CEST    October 12, 2022 2:05:14 PM CEST    Success
Checking Ilom patch Version              October 12, 2022 2:05:14 PM CEST    October 12, 2022 2:05:15 PM CEST    Success
Patch location validation                October 12, 2022 2:05:15 PM CEST    October 12, 2022 2:05:15 PM CEST    Success
Patch location validation                October 12, 2022 2:05:15 PM CEST    October 12, 2022 2:05:15 PM CEST    Success
Save password in Wallet                  October 12, 2022 2:05:15 PM CEST    October 12, 2022 2:05:16 PM CEST    Success
Disabling IPMI v2                        October 12, 2022 2:05:17 PM CEST    October 12, 2022 2:05:19 PM CEST    Success
Apply Ilom patch                         October 12, 2022 2:05:19 PM CEST    October 12, 2022 2:17:07 PM CEST    Success
Apply Ilom patch                         October 12, 2022 2:17:07 PM CEST    October 12, 2022 2:26:49 PM CEST    Success
Copying Flash Bios to Temp location      October 12, 2022 2:26:49 PM CEST    October 12, 2022 2:26:49 PM CEST    Success
Copying Flash Bios to Temp location      October 12, 2022 2:26:49 PM CEST    October 12, 2022 2:26:49 PM CEST    Success
Remove AsrConfBackup File                October 12, 2022 2:26:49 PM CEST    October 12, 2022 2:26:49 PM CEST    Success
Starting the clusterware                 October 12, 2022 2:28:52 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Registering image                        October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Registering working copy                 October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Registering image                        October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Creating GI home directories             October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Extract GI clone                         October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Extract GI clone                         October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Provisioning Software Only GI with RHP   October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:30:39 PM CEST    Success
Patch GI with RHP                        October 12, 2022 2:30:39 PM CEST    October 12, 2022 2:44:42 PM CEST    Success
Updating GIHome version                  October 12, 2022 2:44:43 PM CEST    October 12, 2022 2:44:46 PM CEST    Success
Updating GIHome version                  October 12, 2022 2:44:43 PM CEST    October 12, 2022 2:44:47 PM CEST    Success
Validate GI availability                 October 12, 2022 2:44:50 PM CEST    October 12, 2022 2:44:51 PM CEST    Success
Patch KVM CRS type                       October 12, 2022 2:44:51 PM CEST    October 12, 2022 2:44:51 PM CEST    Success
Patch VM vDisks CRS dependencies         October 12, 2022 2:44:51 PM CEST    October 12, 2022 2:44:51 PM CEST    Success
Update System version                    October 12, 2022 2:44:51 PM CEST    October 12, 2022 2:44:51 PM CEST    Success
Update System version                    October 12, 2022 2:44:51 PM CEST    October 12, 2022 2:44:51 PM CEST    Success
Cleanup JRE Home                         October 12, 2022 2:44:52 PM CEST    October 12, 2022 2:44:52 PM CEST    Success
Add SYSNAME in Env                       October 12, 2022 2:44:52 PM CEST    October 12, 2022 2:44:52 PM CEST    Success
Starting the clusterware                 October 12, 2022 2:44:52 PM CEST    October 12, 2022 2:44:52 PM CEST    Success
Setting ACL for disk groups              October 12, 2022 2:44:52 PM CEST    October 12, 2022 2:45:00 PM CEST    Success
Update previous workarounds              October 12, 2022 2:47:21 PM CEST    October 12, 2022 2:47:21 PM CEST    Success
Update previous workarounds              October 12, 2022 2:47:21 PM CEST    October 12, 2022 2:47:21 PM CEST    Success
PreRebootNode Actions                    October 12, 2022 2:47:21 PM CEST    October 12, 2022 2:48:16 PM CEST    Success
PreRebootNode Actions                    October 12, 2022 2:48:16 PM CEST    October 12, 2022 2:49:07 PM CEST    Success
Reboot Ilom                              October 12, 2022 2:49:07 PM CEST    October 12, 2022 2:49:07 PM CEST    Success
Reboot Ilom                              October 12, 2022 2:49:07 PM CEST    October 12, 2022 2:49:07 PM CEST    Success

Now we have to patch the storage

Let’s run the prepatch report for the storage

[2022-10-12 15:08:54 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport -st -v 19.16.0.0.0

Job details
----------------------------------------------------------------
                     ID:  24fbea1f-8a0c-49bc-b87c-412c64341f62
            Description:  Patch pre-checks for [STORAGE]
                 Status:  Created
                Created:  October 12, 2022 3:09:36 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i 24fbea1f-8a0c-49bc-b87c-412c64341f62' to check details of results

Task Name                                Start Time                          End Time                            Status

Let’s validate the status of the job

[2022-10-12 15:10:22 root@oda-deva]# odacli describe-prepatchreport -i 24fbea1f-8a0c-49bc-b87c-412c64341f62

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  24fbea1f-8a0c-49bc-b87c-412c64341f62
            Description:  Patch pre-checks for [STORAGE]
                 Status:  SUCCESS
                Created:  October 12, 2022 3:09:36 PM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
oda-deva

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate rolling patch          Success   Rolling mode patching allowed as
                                          there is no expander and controller
                                          upgrade.
Validate command execution      Success   Validated command execution

Node Name
---------------
oda-devb

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.16.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate rolling patch          Success   Rolling mode patching allowed as
                                          there is no expander and controller
                                          upgrade.
Validate command execution      Success   Validated command execution

Then let’s patch the storage

[2022-10-12 15:10:28 root@oda-deva]# /opt/oracle/dcs/bin/odacli update-storage -v 19.16.0.0.0 --rolling
{
  "jobId" : "413bb3b5-77b8-44ae-b1b1-ca3d6ea7fc9e",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "October 12, 2022 15:11:10 PM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "October 12, 2022 15:11:10 PM CEST",
  "jobType" : null
}

The operation went fine a few minutes after

[2022-10-12 15:11:20 root@oda-deva]# odacli describe-job -i "413bb3b5-77b8-44ae-b1b1-ca3d6ea7fc9e"

Job details
----------------------------------------------------------------
                     ID:  413bb3b5-77b8-44ae-b1b1-ca3d6ea7fc9e
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  October 12, 2022 3:11:10 PM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           October 12, 2022 3:11:20 PM CEST    October 12, 2022 3:11:26 PM CEST    Success

After the server patch, if you decide to update the dbhomes you can do following steps

First list your dbhomes

[2022-10-12 15:11:51 root@oda-deva]# odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
a1bfe23e-2569-407b-8b87-7af9f9f586bf     OraDB19000_home1     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED

Run the prepatch report for the dbhomes you want to update

[2022-10-12 15:16:51 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport --dbhome --dbhomeid a1bfe23e-2569-407b-8b87-7af9f9f586bf -v 19.16.0.0.0

Job details
----------------------------------------------------------------
                     ID:  7bdce971-fc80-4781-8702-458f6f8d2783
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  Created
                Created:  October 12, 2022 3:16:56 PM CEST
                Message:  Use 'odacli describe-prepatchreport -i 7bdce971-fc80-4781-8702-458f6f8d2783' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

Then validate that there is no error in the prepatch report.

And update the dbhome

# /opt/oracle/dcs/bin/odacli update-dbhome --dbhomeid a1bfe23e-2569-407b-8b87-7af9f9f586bf -v 19.16.0.0.0

The job should return SUCCESS

[2022-10-13 15:40:10 root@oda-devb]# odacli describe-job -i 9ff71b6a-cfb1-494a-9fc2-96e903c293e1

Job details
----------------------------------------------------------------
                     ID:  9ff71b6a-cfb1-494a-9fc2-96e903c293e1
            Description:  DB Home Patching: Home Id is a1bfe23e-2569-407b-8b87-7af9f9f586bf
                 Status:  Success
                Created:  October 13, 2022 6:00:35 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating wallet for DB Client            October 13, 2022 6:01:45 AM CEST    October 13, 2022 6:01:45 AM CEST    Success
Patch databases by RHP                   October 13, 2022 6:01:45 AM CEST    October 13, 2022 6:19:09 AM CEST    Success
Updating database metadata               October 13, 2022 6:19:09 AM CEST    October 13, 2022 6:19:09 AM CEST    Success
Set log_archive_dest for Database        October 13, 2022 6:19:09 AM CEST    October 13, 2022 6:19:13 AM CEST    Success
Patch databases by RHP                   October 13, 2022 6:19:13 AM CEST    October 13, 2022 6:36:09 AM CEST    Success
Updating database metadata               October 13, 2022 6:36:09 AM CEST    October 13, 2022 6:36:09 AM CEST    Success
Set log_archive_dest for Database        October 13, 2022 6:36:09 AM CEST    October 13, 2022 6:36:13 AM CEST    Success
Update System version                    October 13, 2022 6:36:13 AM CEST    October 13, 2022 6:36:13 AM CEST    Success
Update System version                    October 13, 2022 6:36:13 AM CEST    October 13, 2022 6:36:13 AM CEST    Success
TDE parameter update                     October 13, 2022 6:36:45 AM CEST    October 13, 2022 6:36:45 AM CEST    Success

Conclusion

As we can see patching an ODA HA is not different than patching an non-HA ODA. Just some recommendations

-Read the official documentation

-Backup your system before with odabr

-Ensure that you have at least 15G of free space in / /opt and /u01 filesystems