In this blog I am describing an ODA HA (serveroda1 and serveroda2) patch from 19.10 to 19.17. The process is the same that patching a non-HA ODA. Few tasks may need to be done on both servers, otherwise all steps are done from one node. If there are some tasks needed on both nodes, we will specify.

My colleague Jerome already do a nice blog for non-HA ODA.

In the documentation we can find possible versions that can upgraded to version 19.17.

As we are actually on 19.10, we decide to do following path

19.10 ==>19.14 ==>19.17

Required patches

Download following patches and transfer them to one node of the ODA HA

For 19.14

patch 33702951 Oracle Database Appliance Server Patch for the ODACLI/DCS stack

patch 30403673 Oracle Grid Infrastructure clone files 

patch 30403662 Oracle Database clone files

For 19.17

patch 34753059 Oracle Database Appliance Server Patch

patch 30403673 Oracle Grid Infrastructure clone files

patch 30403662 Oracle Database clone files

Below our source configuration

[[email protected] bin]# odacli describe-component
System Version
---------------
19.10.0.0.0

System node Name
---------------
serveroda1

Local System Version
---------------
19.10.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.10.0.0.0           up-to-date

GI                                        19.10.0.0.210119      up-to-date

DB                                        19.10.0.0.210119      up-to-date

DCSAGENT                                  19.10.0.0.0           up-to-date

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    up-to-date

BIOS                                      41080800              up-to-date

FIRMWARECONTROLLER                        16.00.08.00           up-to-date

FIRMWAREEXPANDER                          0310                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             0112                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     A170                  up-to-date
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8 ]
}

HMP                                       2.4.7.0.1             up-to-date

System node Name
---------------
serveroda2

Local System Version
---------------
19.10.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.10.0.0.0           up-to-date

GI                                        19.10.0.0.210119      up-to-date

DB                                        19.10.0.0.210119      up-to-date

DCSAGENT                                  19.10.0.0.0           up-to-date

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    up-to-date

BIOS                                      41080800              up-to-date

FIRMWARECONTROLLER                        16.00.08.00           up-to-date

FIRMWAREEXPANDER                          0310                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             0112                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     A170                  up-to-date
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8 ]
}

HMP                                       2.4.7.0.1             up-to-date

1- Patch from 19.10 to 19.14

Update the repository with the server patch

Unpack the zip files and use following command to update the repository

[[email protected] bin]# odacli update-repository -f /u03/app/oracle/soft/19.14/oda-sm-19.14.0.0.0-220215-server.zip
{
  "jobId" : "f5308fcf-4bf9-42dd-8742-8ade96105968",
  "status" : "Created",
  "message" : "/u03/app/oracle/soft/19.14/oda-sm-19.14.0.0.0-220215-server.zip",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 08:48:14 AM CET",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "January 03, 2023 08:48:14 AM CET"
}

Validate that the job was fine

[[email protected] bin]# odacli describe-job -i "f5308fcf-4bf9-42dd-8742-8ade96105968"

At this moment we can again check the status of the components

[[email protected] bin]# odacli describe-component
System Version
---------------
19.10.0.0.0

System node Name
---------------
serveroda1

Local System Version
---------------
19.10.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.10.0.0.0           19.14.0.0.0

GI                                        19.10.0.0.210119      19.14.0.0.220118

DB                                        19.10.0.0.210119      19.14.0.0.220118

DCSAGENT                                  19.10.0.0.0           19.14.0.0.0

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    5.0.2.24.r141466

BIOS                                      41080800              41100500

FIRMWARECONTROLLER                        16.00.08.00           up-to-date

FIRMWAREEXPANDER                          0310                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             0112                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     A170                  up-to-date
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8 ]
}

HMP                                       2.4.7.0.1             2.4.8.0.600

System node Name
---------------
serveroda2

Local System Version
---------------
19.10.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.10.0.0.0           19.14.0.0.0

GI                                        19.10.0.0.210119      19.14.0.0.220118

DB                                        19.10.0.0.210119      19.14.0.0.220118

DCSAGENT                                  19.10.0.0.0           19.14.0.0.0

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    5.0.2.24.r141466

BIOS                                      41080800              41100500

FIRMWARECONTROLLER                        16.00.08.00           up-to-date

FIRMWAREEXPANDER                          0310                  up-to-date

FIRMWAREDISK {
[ c0d0,c0d1 ]                             0112                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,     A170                  up-to-date
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8 ]
}

HMP                                       2.4.7.0.1             2.4.8.0.600

Update the dcsadmin

[[email protected] bin]# odacli update-dcsadmin -v 19.14.0.0.0
{
  "jobId" : "e36e0cba-f853-4c84-adeb-67660236a858",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 08:51:10 AM CET",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "January 03, 2023 08:51:10 AM CET"
}

Validate the job status

[email protected] bin]# odacli describe-job -i "e36e0cba-f853-4c84-adeb-67660236a858"

Job details
----------------------------------------------------------------
                     ID:  e36e0cba-f853-4c84-adeb-67660236a858
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  January 3, 2023 8:51:10 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                January 3, 2023 8:51:11 AM CET      January 3, 2023 8:51:11 AM CET      Success
Patch location validation                January 3, 2023 8:51:11 AM CET      January 3, 2023 8:51:11 AM CET      Success
dcs-admin upgrade                        January 3, 2023 8:51:11 AM CET      January 3, 2023 8:51:17 AM CET      Success
dcs-admin upgrade                        January 3, 2023 8:51:17 AM CET      January 3, 2023 8:51:24 AM CET      Success

Update the dcscomponents

[[email protected] bin]# odacli update-dcscomponents -v 19.14.0.0.0
{
  "jobId" : "b057eaa3-eaaf-49a7-b662-6c53eb95b3c0",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  ",
  "reports" : null,
  "createTimestamp" : "January 03, 2023 08:52:31 AM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "January 03, 2023 08:56:11 AM CET"
}

It’s normal that the describe-job for this job does not return any value. You just should have SUCCESS on the status for the command above

[[email protected] bin]# odacli describe-job -i "b057eaa3-eaaf-49a7-b662-6c53eb95b3c0"
DCS-10000:Resource Job with ID b057eaa3-eaaf-49a7-b662-6c53eb95b3c0 is not found.

Update the dcsagent

[[email protected] bin]# odacli update-dcsagent -v 19.14.0.0.0
{
  "jobId" : "e1511288-b202-42c5-bbe8-5c13f4b1cfa9",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 08:57:54 AM CET",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "January 03, 2023 08:57:54 AM CET"
}

The following command should return SUCCESS

[[email protected] bin]# odacli describe-job -i "e1511288-b202-42c5-bbe8-5c13f4b1cfa9"

Update the repository with the 19.14 GI and DB clones

GI clone

[[email protected] bin]#  odacli update-repository -f /u03/app/oracle/soft/19.14/odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip
{
  "jobId" : "b05f15ed-2513-47d3-864f-fae03394c0a5",
  "status" : "Created",
  "message" : "/u03/app/oracle/soft/19.14/odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 09:02:54 AM CET",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "January 03, 2023 09:02:54 AM CET"
}

Validate the status of the job

[[email protected] bin]# odacli describe-job -i  "b05f15ed-2513-47d3-864f-fae03394c0a5"

Job details
----------------------------------------------------------------
                     ID:  b05f15ed-2513-47d3-864f-fae03394c0a5
            Description:  Repository Update
                 Status:  Success
                Created:  January 3, 2023 9:02:54 AM CET
                Message:  /u03/app/oracle/soft/19.14/odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip


Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     January 3, 2023 9:03:06 AM CET      January 3, 2023 9:03:06 AM CET      Success
Setting up ssh equivalance               January 3, 2023 9:03:06 AM CET      January 3, 2023 9:03:06 AM CET      Success
Copy BundleFile                          January 3, 2023 9:03:07 AM CET      January 3, 2023 9:03:47 AM CET      Success
Validating CopiedFile                    January 3, 2023 9:03:47 AM CET      January 3, 2023 9:03:58 AM CET      Success
Unzip bundle                             January 3, 2023 9:03:58 AM CET      January 3, 2023 9:05:35 AM CET      Success
Unzip bundle                             January 3, 2023 9:05:35 AM CET      January 3, 2023 9:07:04 AM CET      Success
Removing ssh keys                        January 3, 2023 9:07:04 AM CET      January 3, 2023 9:07:05 AM CET      Success

DB clone

[[email protected] bin]# odacli update-repository -f /u03/app/oracle/soft/19.14/odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip
{
  "jobId" : "20b39c38-bf4f-4815-9223-fffa8ee06b4f",
  "status" : "Created",
  "message" : "/u03/app/oracle/soft/19.14/odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 09:08:42 AM CET",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "January 03, 2023 09:08:42 AM CET"
}

Validate the job

[[email protected] bin]# odacli describe-job -i "20b39c38-bf4f-4815-9223-fffa8ee06b4f"

Job details
----------------------------------------------------------------
                     ID:  20b39c38-bf4f-4815-9223-fffa8ee06b4f
            Description:  Repository Update
                 Status:  Success
                Created:  January 3, 2023 9:08:42 AM CET
                Message:  /u03/app/oracle/soft/19.14/odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     January 3, 2023 9:08:51 AM CET      January 3, 2023 9:08:51 AM CET      Success
Setting up ssh equivalance               January 3, 2023 9:08:51 AM CET      January 3, 2023 9:08:51 AM CET      Success
Copy BundleFile                          January 3, 2023 9:08:52 AM CET      January 3, 2023 9:09:21 AM CET      Success
Validating CopiedFile                    January 3, 2023 9:09:22 AM CET      January 3, 2023 9:09:31 AM CET      Success
Unzip bundle                             January 3, 2023 9:09:31 AM CET      January 3, 2023 9:10:37 AM CET      Success
Unzip bundle                             January 3, 2023 9:10:37 AM CET      January 3, 2023 9:11:43 AM CET      Success
Removing ssh keys                        January 3, 2023 9:11:43 AM CET      January 3, 2023 9:11:44 AM CET      Success

Generate a prepatch report

[[email protected] bin]# odacli create-prepatchreport -s -v 19.14.0.0.0

Job details
----------------------------------------------------------------
                     ID:  d05aa323-7ce8-4393-a646-9e6052fc2e12
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  Created
                Created:  January 3, 2023 9:20:33 AM CET
                Message:  Use 'odacli describe-prepatchreport -i d05aa323-7ce8-4393-a646-9e6052fc2e12' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

Check If there is any error. Any failed step should be corrected

[[email protected] bin]# odacli describe-prepatchreport -i d05aa323-7ce8-4393-a646-9e6052fc2e12

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d05aa323-7ce8-4393-a646-9e6052fc2e12
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  January 3, 2023 9:20:33 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
serveroda1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Node Name
---------------
serveroda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Update the server now

As there is no error, we can now launch the  server patching

[[email protected] bin]#  /opt/oracle/dcs/bin/odacli update-server -v 19.14.0.0.0
{
  "jobId" : "aa4ffe45-eae2-4eca-8cd0-f4c0b9ae70f8",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 09:38:09 AM CET",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "January 03, 2023 09:38:09 AM CET"
}

And the end the result should be SUCCESS

[[email protected] bin]# odacli describe-job -i "aa4ffe45-eae2-4eca-8cd0-f4c0b9ae70f8"

Job details
----------------------------------------------------------------
                     ID:  aa4ffe45-eae2-4eca-8cd0-f4c0b9ae70f8
            Description:  Server Patching
                 Status:  Success
                Created:  January 3, 2023 9:38:09 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              January 3, 2023 9:38:30 AM CET      January 3, 2023 9:38:30 AM CET      Success
Validating GI user metadata              January 3, 2023 9:38:30 AM CET      January 3, 2023 9:38:30 AM CET      Success
Validate ILOM server reachable           January 3, 2023 9:38:30 AM CET      January 3, 2023 9:38:30 AM CET      Success
Validate ILOM server reachable           January 3, 2023 9:38:30 AM CET      January 3, 2023 9:38:30 AM CET      Success
Configure export clones resource         January 3, 2023 9:40:48 AM CET      January 3, 2023 9:40:50 AM CET      Success
Creating repositories using yum          January 3, 2023 9:40:56 AM CET      January 3, 2023 9:41:02 AM CET      Success
Updating YumPluginVersionLock rpm        January 3, 2023 9:41:02 AM CET      January 3, 2023 9:41:02 AM CET      Success
Applying OS Patches                      January 3, 2023 9:41:02 AM CET      January 3, 2023 9:46:37 AM CET      Success
Creating repositories using yum          January 3, 2023 9:46:54 AM CET      January 3, 2023 9:46:54 AM CET      Success
Applying HMP Patches                     January 3, 2023 9:46:54 AM CET      January 3, 2023 9:47:10 AM CET      Success
Patch location validation                January 3, 2023 9:47:12 AM CET      January 3, 2023 9:47:12 AM CET      Success
Patch location validation                January 3, 2023 9:47:12 AM CET      January 3, 2023 9:47:12 AM CET      Success
oda-hw-mgmt upgrade                      January 3, 2023 9:47:12 AM CET      January 3, 2023 9:47:45 AM CET      Success
oda-hw-mgmt upgrade                      January 3, 2023 9:47:45 AM CET      January 3, 2023 9:48:16 AM CET      Success
OSS Patching                             January 3, 2023 9:48:16 AM CET      January 3, 2023 9:48:17 AM CET      Success
Applying Firmware Disk Patches           January 3, 2023 9:48:21 AM CET      January 3, 2023 9:48:24 AM CET      Success
Applying Firmware Controller Patches     January 3, 2023 9:48:28 AM CET      January 3, 2023 9:48:32 AM CET      Success
Checking Ilom patch Version              January 3, 2023 9:48:32 AM CET      January 3, 2023 9:48:32 AM CET      Success
Checking Ilom patch Version              January 3, 2023 9:48:32 AM CET      January 3, 2023 9:48:32 AM CET      Success
Patch location validation                January 3, 2023 9:48:33 AM CET      January 3, 2023 9:48:33 AM CET      Success
Patch location validation                January 3, 2023 9:48:33 AM CET      January 3, 2023 9:48:33 AM CET      Success
Save password in Wallet                  January 3, 2023 9:48:33 AM CET      January 3, 2023 9:48:34 AM CET      Success
Disabling IPMI v2                        January 3, 2023 9:48:36 AM CET      January 3, 2023 9:48:37 AM CET      Success
Apply Ilom patch                         January 3, 2023 9:48:37 AM CET      January 3, 2023 9:57:18 AM CET      Success
Apply Ilom patch                         January 3, 2023 9:57:18 AM CET      January 3, 2023 10:05:31 AM CET     Success
Copying Flash Bios to Temp location      January 3, 2023 10:05:31 AM CET     January 3, 2023 10:05:31 AM CET     Success
Copying Flash Bios to Temp location      January 3, 2023 10:05:31 AM CET     January 3, 2023 10:05:31 AM CET     Success
Adding USER SSH_EQUIVALENCE              January 3, 2023 10:05:32 AM CET     January 3, 2023 10:05:33 AM CET     Success
Adding USER SSH_EQUIVALENCE              January 3, 2023 10:05:33 AM CET     January 3, 2023 10:05:34 AM CET     Success
Adding USER SSH_EQUIVALENCE              January 3, 2023 10:05:34 AM CET     January 3, 2023 10:05:35 AM CET     Success
Starting the clusterware                 January 3, 2023 10:07:56 AM CET     January 3, 2023 10:10:10 AM CET     Success
registering image                        January 3, 2023 10:10:12 AM CET     January 3, 2023 10:10:13 AM CET     Success
registering working copy                 January 3, 2023 10:10:14 AM CET     January 3, 2023 10:10:14 AM CET     Success
registering image                        January 3, 2023 10:10:14 AM CET     January 3, 2023 10:10:16 AM CET     Success
Creating GI home directories             January 3, 2023 10:10:17 AM CET     January 3, 2023 10:10:20 AM CET     Success
Extract GI clone                         January 3, 2023 10:10:21 AM CET     January 3, 2023 10:10:21 AM CET     Success
Extract GI clone                         January 3, 2023 10:10:21 AM CET     January 3, 2023 10:10:21 AM CET     Success
Provisioning Software Only GI with RHP   January 3, 2023 10:10:22 AM CET     January 3, 2023 10:10:22 AM CET     Success
Patch GI with RHP                        January 3, 2023 10:10:22 AM CET     January 3, 2023 10:26:57 AM CET     Success
Updating GIHome version                  January 3, 2023 10:26:59 AM CET     January 3, 2023 10:27:06 AM CET     Success
Updating GIHome version                  January 3, 2023 10:26:59 AM CET     January 3, 2023 10:27:03 AM CET     Success
Validate GI availability                 January 3, 2023 10:27:14 AM CET     January 3, 2023 10:27:14 AM CET     Success
Patch KVM CRS type                       January 3, 2023 10:27:14 AM CET     January 3, 2023 10:27:14 AM CET     Success
Update System version                    January 3, 2023 10:27:14 AM CET     January 3, 2023 10:27:14 AM CET     Success
Update System version                    January 3, 2023 10:27:14 AM CET     January 3, 2023 10:27:14 AM CET     Success
Cleanup JRE Home                         January 3, 2023 10:27:14 AM CET     January 3, 2023 10:27:15 AM CET     Success
Add SYSNAME in Env                       January 3, 2023 10:27:15 AM CET     January 3, 2023 10:27:15 AM CET     Success
Starting the clusterware                 January 3, 2023 10:27:15 AM CET     January 3, 2023 10:27:15 AM CET     Success
Setting ACL for disk groups              January 3, 2023 10:27:15 AM CET     January 3, 2023 10:27:22 AM CET     Success
Update previous workarounds              January 3, 2023 10:29:12 AM CET     January 3, 2023 10:29:12 AM CET     Success
Update previous workarounds              January 3, 2023 10:29:12 AM CET     January 3, 2023 10:29:12 AM CET     Success
preRebootNode Actions                    January 3, 2023 10:29:12 AM CET     January 3, 2023 10:30:16 AM CET     Success
preRebootNode Actions                    January 3, 2023 10:30:16 AM CET     January 3, 2023 10:31:09 AM CET     Success
Reboot Ilom                              January 3, 2023 10:31:10 AM CET     January 3, 2023 10:31:10 AM CET     Success
Reboot Ilom                              January 3, 2023 10:31:10 AM CET     January 3, 2023 10:31:10 AM CET     Success

Validate the new components

[[email protected] bin]# odacli describe-component
System Version
---------------
19.14.0.0.0

System node Name
---------------
serveroda1

Local System Version
---------------
19.14.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date

GI
                                          19.14.0.0.220118      up-to-date

DB
                                          19.10.0.0.210119      19.14.0.0.220118

DCSCONTROLLER
                                          19.14.0.0.0           up-to-date

DCSCLI
                                          19.14.0.0.0           up-to-date

DCSAGENT
                                          19.14.0.0.0           up-to-date

DCSADMIN
                                          19.14.0.0.0           up-to-date

OS
                                          7.9                   up-to-date

ILOM
                                          5.0.2.24.r141466      up-to-date

BIOS
                                          41100500              up-to-date

SHARED CONTROLLER FIRMWARE
                                          16.00.08.00           up-to-date

FIRMWAREEXPANDER
                                          0310                  up-to-date

LOCAL DISK FIRMWARE
                                          0112                  up-to-date

SHARED DISK FIRMWARE {
[ c0d0,c0d1 ]
                                          0112                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8 ]                          A170                  up-to-date
}

HMP
                                          2.4.8.0.600           up-to-date

System node Name
---------------
serveroda2

Local System Version
---------------
19.14.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date

GI
                                          19.14.0.0.220118      up-to-date

DB
                                          19.10.0.0.210119      19.14.0.0.220118

DCSCONTROLLER
                                          19.14.0.0.0           up-to-date

DCSCLI
                                          19.14.0.0.0           up-to-date

DCSAGENT
                                          19.14.0.0.0           up-to-date

DCSADMIN
                                          19.14.0.0.0           up-to-date

OS
                                          7.9                   up-to-date

ILOM
                                          5.0.2.24.r141466      up-to-date

BIOS
                                          41100500              up-to-date

SHARED CONTROLLER FIRMWARE
                                          16.00.08.00           up-to-date

FIRMWAREEXPANDER
                                          0310                  up-to-date

LOCAL DISK FIRMWARE
                                          0112                  up-to-date

SHARED DISK FIRMWARE {
[ c0d0,c0d1 ]
                                          0112                  up-to-date
[ c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8 ]                          A170                  up-to-date
}

HMP
                                          2.4.8.0.600           up-to-date

Update the storage components

Run a prepatch report for storage

[email protected] bin]# /opt/oracle/dcs/bin/odacli create-prepatchreport -st -v 19.14.0.0.0

Job details
----------------------------------------------------------------
                     ID:  3197445b-00a6-44a7-92c4-eded9dd0ed8a
            Description:  Patch pre-checks for [STORAGE]
                 Status:  Created
                Created:  January 3, 2023 10:51:44 AM CET
                Message:  Use 'odacli describe-prepatchreport -i 3197445b-00a6-44a7-92c4-eded9dd0ed8a' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- 

Validate that there is no error

[[email protected] bin]# odacli describe-prepatchreport -i 3197445b-00a6-44a7-92c4-eded9dd0ed8a

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  3197445b-00a6-44a7-92c4-eded9dd0ed8a
            Description:  Patch pre-checks for [STORAGE]
                 Status:  SUCCESS
                Created:  January 3, 2023 10:51:44 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
serveroda1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate rolling patch          Success   Rolling mode patching allowed as
                                          there is no expander and controller
                                          upgrade.
Validate command execution      Success   Validated command execution

Node Name
---------------
serveroda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__STORAGE__
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Patch location validation       Success   Verified patch location
Patch tag validation            Success   Verified patch tag
Storage patch tag validation    Success   Verified storage patch location
Verify ASM disks status         Success   ASM disks are online
Validate rolling patch          Success   Rolling mode patching allowed as
                                          there is no expander and controller
                                          upgrade.
Validate command execution      Success   Validated command execution

Now update the storage

[[email protected] bin]# odacli update-storage -v 19.14.0.0.0 --rolling
{
  "jobId" : "860f95b3-f1dd-402a-897d-ebf0ca9b4584",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 10:57:20 AM CET",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "January 03, 2023 10:57:20 AM CET"

Validate the job status

[[email protected] bin]# odacli describe-job -i 860f95b3-f1dd-402a-897d-ebf0ca9b4584

Configure dbhome storage

Starting with Oracle Database Appliance release 19.11, Oracle Database homes are not created on the local disk. Oracle Database homes are created on an Oracle ACFS-managed file system. As we are patching from 19.10, we have to configure a dbhome storage. In fact we decide on which disk group future dbhomes will reside

[[email protected] bin]# odacli configure-dbhome-storage -dg DATA
{
  "jobId" : "5e33c0e5-64c0-404e-a3b9-5ca061830b86",
  "status" : "Created",
  "message" : "DBhome storage configuration is stored in the metadata and will take effect on next database home creation.",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 11:03:00 AM CET",
  "resourceList" : [ ],
  "description" : "Configure database home storage",
  "updatedTime" : "January 03, 2023 11:03:00 AM CET"
}

Validate the job

[[email protected] bin]# odacli describe-job -i  "5e33c0e5-64c0-404e-a3b9-5ca061830b86"

Job details
----------------------------------------------------------------
                     ID:  5e33c0e5-64c0-404e-a3b9-5ca061830b86
            Description:  Configure database home storage
                 Status:  Success
                Created:  January 3, 2023 11:03:00 AM CET
                Message:  DBhome storage configuration is stored in the metadata and will take effect on next dbhome creation.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Configuring database home storage        January 3, 2023 11:03:00 AM CET     January 3, 2023 11:03:00 AM CET     Success

Update dbhome

In our case we decide to only update the oracle 19c homes

[[email protected] ~]# odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
968b790a-9193-4450-884b-04b1e53fdf2d     OraDB19000_home1     19.10.0.0.210119                         /u01/app/oracle/product/19.0.0.0/dbhome_1     CONFIGURED
21e5db93-d0e8-4e47-bcb1-5da8876e8d24     OraDB19000_home2     19.10.0.0.210119                         /u01/app/oracle/product/19.0.0.0/dbhome_2     CONFIGURED

Run a prepatch report for the dbhome to be updated

[[email protected] ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport --dbhome --dbhomeid 968b790a-9193-4450-884b-04b1e53fdf2d -v 19.14.0.0.0

Job details
----------------------------------------------------------------
                     ID:  ab290e9f-2647-4302-9d1b-e0887d29e254
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  Created
                Created:  January 3, 2023 11:17:10 AM CET
                Message:  Use 'odacli describe-prepatchreport -i ab290e9f-2647-4302-9d1b-e0887d29e254' to check details of results

Check that the report is fine

[[email protected] bin]# odacli describe-prepatchreport -i ab290e9f-2647-4302-9d1b-e0887d29e254

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  ab290e9f-2647-4302-9d1b-e0887d29e254
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  SUCCESS
                Created:  January 3, 2023 11:17:10 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
serveroda1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          968b790a-9193-4450-884b-04b1e53fdf2d
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Space for /u01/app/odaorahome will be
db                                        created during patching
Validate dbHomesOnACFS          Success   User has configured diskgroup for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Node Name
---------------
serveroda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          968b790a-9193-4450-884b-04b1e53fdf2d
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Space for /u01/app/odaorahome will be
db                                        created during patching
Validate dbHomesOnACFS          Success   User has configured diskgroup for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

And then update the dbhome

[[email protected] bin]# /opt/oracle/dcs/bin/odacli update-dbhome --dbhomeid 968b790a-9193-4450-884b-04b1e53fdf2d -v 19.14.0.0.0
{
  "jobId" : "eb8fa19d-60a1-4f45-88e1-c95b4a059417",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 11:40:23 AM CET",
  "resourceList" : [ ],
  "description" : "DB Home Patching: Home Id is 968b790a-9193-4450-884b-04b1e53fdf2d",
  "updatedTime" : "January 03, 2023 11:40:24 AM CET"
}

The job should return SUCCESS

[[email protected] bin]# odacli describe-job -i  "eb8fa19d-60a1-4f45-88e1-c95b4a059417"

Job details
----------------------------------------------------------------
                     ID:  eb8fa19d-60a1-4f45-88e1-c95b4a059417
            Description:  DB Home Patching: Home Id is 968b790a-9193-4450-884b-04b1e53fdf2d
                 Status:  Success
                Created:  January 3, 2023 11:40:23 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Adding USER SSH_EQUIVALENCE              January 3, 2023 11:40:37 AM CET     January 3, 2023 11:40:38 AM CET     Success
Adding USER SSH_EQUIVALENCE              January 3, 2023 11:40:38 AM CET     January 3, 2023 11:40:39 AM CET     Success
Adding USER SSH_EQUIVALENCE              January 3, 2023 11:40:39 AM CET     January 3, 2023 11:40:40 AM CET     Success
Creating wallet for DB Client            January 3, 2023 11:41:20 AM CET     January 3, 2023 11:41:20 AM CET     Success
Patch databases by RHP                   January 3, 2023 11:41:20 AM CET     January 3, 2023 11:43:16 AM CET     Success
updating database metadata               January 3, 2023 11:44:10 AM CET     January 3, 2023 11:44:10 AM CET     Success
Set log_archive_dest for Database        January 3, 2023 11:44:12 AM CET     January 3, 2023 11:44:13 AM CET     Success
Patch databases by RHP                   January 3, 2023 11:44:13 AM CET     January 3, 2023 11:51:35 AM CET     Success
updating database metadata               January 3, 2023 11:52:44 AM CET     January 3, 2023 11:52:44 AM CET     Success
Set log_archive_dest for Database        January 3, 2023 11:52:45 AM CET     January 3, 2023 11:52:47 AM CET     Success
Patch databases by RHP                   January 3, 2023 11:52:47 AM CET     January 3, 2023 11:59:52 AM CET     Success
updating database metadata               January 3, 2023 12:01:06 PM CET     January 3, 2023 12:01:06 PM CET     Success
Set log_archive_dest for Database        January 3, 2023 12:01:08 PM CET     January 3, 2023 12:01:11 PM CET     Success
Patch databases by RHP                   January 3, 2023 12:01:11 PM CET     January 3, 2023 12:02:49 PM CET     Success
updating database metadata               January 3, 2023 12:03:43 PM CET     January 3, 2023 12:03:43 PM CET     Success
Set log_archive_dest for Database        January 3, 2023 12:03:46 PM CET     January 3, 2023 12:03:47 PM CET     Success
Patch databases by RHP                   January 3, 2023 12:03:47 PM CET     January 3, 2023 12:10:54 PM CET     Success
updating database metadata               January 3, 2023 12:12:01 PM CET     January 3, 2023 12:12:01 PM CET     Success
Set log_archive_dest for Database        January 3, 2023 12:12:02 PM CET     January 3, 2023 12:12:05 PM CET     Success
Patch databases by RHP                   January 3, 2023 12:12:05 PM CET     January 3, 2023 12:13:45 PM CET     Success
updating database metadata               January 3, 2023 12:14:38 PM CET     January 3, 2023 12:14:38 PM CET     Success
Set log_archive_dest for Database        January 3, 2023 12:14:41 PM CET     January 3, 2023 12:14:42 PM CET     Success
Patch databases by RHP                   January 3, 2023 12:14:42 PM CET     January 3, 2023 12:21:49 PM CET     Success
updating database metadata               January 3, 2023 12:22:57 PM CET     January 3, 2023 12:22:57 PM CET     Success
Set log_archive_dest for Database        January 3, 2023 12:22:59 PM CET     January 3, 2023 12:23:02 PM CET     Success
Patch databases by RHP                   January 3, 2023 12:23:02 PM CET     January 3, 2023 12:30:09 PM CET     Success
updating database metadata               January 3, 2023 12:31:15 PM CET     January 3, 2023 12:31:15 PM CET     Success
Set log_archive_dest for Database        January 3, 2023 12:31:16 PM CET     January 3, 2023 12:31:19 PM CET     Success
Update System version                    January 3, 2023 12:31:19 PM CET     January 3, 2023 12:31:19 PM CET     Success
Update System version                    January 3, 2023 12:31:19 PM CET     January 3, 2023 12:31:19 PM CET     Success
TDE parameter update                     January 3, 2023 12:32:03 PM CET     January 3, 2023 12:32:03 PM CET     Success

For each database of the updated dbhome run the following command

SQL> alter system set "_enable_numa_support"=true scope=spfile;

2- Patch from 19.10 to 19.14

The steps are the same. Some ouputs below are truncated

Update the repository with the server patch and validate that the job is fine

[[email protected] bin]# /opt/oracle/dcs/bin/odacli update-repository -f /u03/app/oracle/soft/19.17/oda-sm-19.17.0.0.0-221126.1-server.zip
{
  "jobId" : "45dedaa4-0287-4e4b-843c-0b2cd57b48b7",
  "status" : "Created",
  "message" : "/u03/app/oracle/soft/19.17/oda-sm-19.17.0.0.0-221126.1-server.zip",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 12:47:13 PM CET",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "January 03, 2023 12:47:13 PM CET"
}
[[email protected] bin]# odacli describe-job -i  "45dedaa4-0287-4e4b-843c-0b2cd57b48b7"

Job details
----------------------------------------------------------------
                     ID:  45dedaa4-0287-4e4b-843c-0b2cd57b48b7
            Description:  Repository Update
                 Status:  Success
                Created:  January 3, 2023 12:47:13 PM CET
                Message:  /u03/app/oracle/soft/19.17/oda-sm-19.17.0.0.0-221126.1-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace                     January 3, 2023 12:47:21 PM CET     January 3, 2023 12:47:21 PM CET     Success
...
...
January 3, 2023 12:48:12 PM CET     January 3, 2023 12:48:42 PM CET     Success
Removing ssh keys                        January 3, 2023 12:48:44 PM CET     January 3, 2023 12:48:45 PM CET     Success

Update the DscAdmin

[[email protected] bin]# odacli update-dcsadmin -v 19.17.0.0.0
{
  "jobId" : "32c630b5-d796-42e0-94a2-9900c8ce2886",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 12:52:14 PM CET",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "January 03, 2023 12:52:14 PM CET"
}
[[email protected] bin]# odacli describe-job -i "32c630b5-d796-42e0-94a2-9900c8ce2886"

Job details
----------------------------------------------------------------
                     ID:  32c630b5-d796-42e0-94a2-9900c8ce2886
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  January 3, 2023 12:52:14 PM CET
                Message:
...
...

Update the DCS components

[[email protected] bin]# odacli update-dcscomponents -v 19.17.0.0.0
{
  "jobId" : "aed5c87c-d974-4942-baea-80b524436c45",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  ",
  "reports" : null,
  "createTimestamp" : "January 03, 2023 12:53:19 PM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "January 03, 2023 12:56:49 PM CET"
}

Update the DCS agent

[[email protected] bin]# odacli update-dcsagent -v 19.17.0.0.0
{
  "jobId" : "c401c1c6-cb7f-4ac8-8b91-31b91ddf429c",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 12:58:04 PM CET",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "January 03, 2023 12:58:04 PM CET",
  "jobType" : null
}
[[email protected] bin]# odacli describe-job -i "c401c1c6-cb7f-4ac8-8b91-31b91ddf429c"

Job details
----------------------------------------------------------------
                     ID:  c401c1c6-cb7f-4ac8-8b91-31b91ddf429c
            Description:  DcsAgent patching
                 Status:  Success
                Created:  January 3, 2023 12:58:04 PM CET
                Message:

...
...

Update the repository with the GI  and DB clones

GI

[[email protected] bin]# odacli update-repository -f /u03/app/oracle/soft/19.17/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip
{
  "jobId" : "a6c09e2e-f3ee-45a7-9b88-e0dcd85df560",
  "status" : "Created",
  "message" : "/u03/app/oracle/soft/19.17/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 13:04:16 PM CET",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "January 03, 2023 13:04:16 PM CET",
  "jobType" : null
}
[[email protected] bin]# odacli describe-job -i "a6c09e2e-f3ee-45a7-9b88-e0dcd85df560"

Job details
----------------------------------------------------------------
                     ID:  a6c09e2e-f3ee-45a7-9b88-e0dcd85df560
            Description:  Repository Update
                 Status:  Success
                Created:  January 3, 2023 1:04:16 PM CET
                Message:  /u03/app/oracle/soft/19.17/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip
...
...

DB

[[email protected] bin]# odacli update-repository -f /u03/app/oracle/soft/19.17/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip
{
  "jobId" : "6d660d44-78c5-4f52-98cd-c7e25b734c65",
  "status" : "Created",
  "message" : "/u03/app/oracle/soft/19.17/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 13:07:00 PM CET",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "January 03, 2023 13:07:00 PM CET",
  "jobType" : null
}
[[email protected] bin]# odacli describe-job -i "6d660d44-78c5-4f52-98cd-c7e25b734c65"

Job details
----------------------------------------------------------------
                     ID:  6d660d44-78c5-4f52-98cd-c7e25b734c65
            Description:  Repository Update
                 Status:  Success
                Created:  January 3, 2023 1:07:00 PM CET
                Message:  /u03/app/oracle/soft/19.17/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip

...
...

Run prepatch report for the server

[[email protected] bin]# odacli create-prepatchreport -s -v 19.17.0.0.0

Job details
----------------------------------------------------------------
                     ID:  03a5816c-fa51-42cf-9445-7ee429445c48
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  Created
                Created:  January 3, 2023 1:10:25 PM CET
                Message:  Use 'odacli describe-prepatchreport -i 03a5816c-fa51-42cf-9445-7ee429445c48' to check details of results

Check the results for any error and correct it

[[email protected] bin]# 'odacli describe-prepatchreport -i 03a5816c-fa51-42cf-9445-7ee429445c48

Update the server

[[email protected] bin]# odacli update-server -v 19.17.0.0.0
{
  "jobId" : "57d58ccd-b570-419f-b0b5-d25d58f5adec",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 13:43:29 PM CET",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "January 03, 2023 13:43:29 PM CET",

Validate the results

[[email protected] bin]# odacli describe-job -i "57d58ccd-b570-419f-b0b5-d25d58f5adec"

Job details
----------------------------------------------------------------
                     ID:  57d58ccd-b570-419f-b0b5-d25d58f5adec
            Description:  Server Patching
                 Status:  Success
                Created:  January 3, 2023 1:43:29 PM CET
                Message:  Successfully patched GI with RHP

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              serveroda1                  January 3, 2023 1:43:52 PM CET      January 3, 2023 1:43:52 PM CET      Success
Validating GI user metadata              serveroda2                  January 3, 2023 1:43:52 PM CET      January 3, 2023 1:43:52 PM CET      Success
Validate ILOM server reachable           serveroda2                  January 3, 2023 1:43:52 PM CET      January 3, 2023 1:43:52 PM CET      Success
Validate ILOM server reachable           serveroda1                  January 3, 2023 1:43:52 PM CET      January 3, 2023 1:43:53 PM CET      Success
Validate DCS Admin mTLS setup            serveroda2                  January 3, 2023 1:43:53 PM CET      January 3, 2023 1:43:53 PM CET      Success
Validate DCS Admin mTLS setup            serveroda1                  January 3, 2023 1:43:53 PM CET      January 3, 2023 1:43:53 PM CET      Success
Configure export clones resource         serveroda1                  January 3, 2023 1:43:55 PM CET      January 3, 2023 1:43:55 PM CET      Success
Creating repositories using yum          serveroda2                  January 3, 2023 1:43:59 PM CET      January 3, 2023 1:44:03 PM CET      Success
Updating YumPluginVersionLock rpm        serveroda2                  January 3, 2023 1:44:03 PM CET      January 3, 2023 1:44:03 PM CET      Success
Applying OS Patches                      serveroda2                  January 3, 2023 1:44:03 PM CET      January 3, 2023 1:55:38 PM CET      Success
Creating repositories using yum          serveroda2                  January 3, 2023 1:55:39 PM CET      January 3, 2023 1:55:39 PM CET      Success
Applying HMP Patches                     serveroda2                  January 3, 2023 1:55:39 PM CET      January 3, 2023 1:55:58 PM CET      Success
Patch location validation                serveroda2                  January 3, 2023 1:55:59 PM CET      January 3, 2023 1:55:59 PM CET      Success
Patch location validation                serveroda1                  January 3, 2023 1:55:59 PM CET      January 3, 2023 1:55:59 PM CET      Success
Oda-hw-mgmt upgrade                      serveroda1                  January 3, 2023 1:55:59 PM CET      January 3, 2023 1:56:33 PM CET      Success
Oda-hw-mgmt upgrade                      serveroda2                  January 3, 2023 1:56:33 PM CET      January 3, 2023 1:57:06 PM CET      Success
OSS Patching                             serveroda2                  January 3, 2023 1:57:06 PM CET      January 3, 2023 1:57:07 PM CET      Success
Applying Firmware Disk Patches           serveroda2                  January 3, 2023 1:57:09 PM CET      January 3, 2023 1:57:12 PM CET      Success
Applying Firmware Controller Patches     serveroda2                  January 3, 2023 1:58:52 PM CET      January 3, 2023 2:00:33 PM CET      Success
Checking Ilom patch Version              serveroda1                  January 3, 2023 2:00:33 PM CET      January 3, 2023 2:00:33 PM CET      Success
Checking Ilom patch Version              serveroda2                  January 3, 2023 2:00:33 PM CET      January 3, 2023 2:00:34 PM CET      Success
Patch location validation                serveroda1                  January 3, 2023 2:00:34 PM CET      January 3, 2023 2:00:34 PM CET      Success
Patch location validation                serveroda2                  January 3, 2023 2:00:34 PM CET      January 3, 2023 2:00:34 PM CET      Success
Save password in Wallet                  serveroda2                  January 3, 2023 2:00:34 PM CET      January 3, 2023 2:00:35 PM CET      Success
Disabling IPMI v2                        serveroda2                  January 3, 2023 2:00:36 PM CET      January 3, 2023 2:00:36 PM CET      Success
Apply Ilom patch                         serveroda1                  January 3, 2023 2:00:36 PM CET      January 3, 2023 2:09:54 PM CET      Success
Apply Ilom patch                         serveroda2                  January 3, 2023 2:09:55 PM CET      January 3, 2023 2:18:47 PM CET      Success
Copying Flash Bios to Temp location      serveroda1                  January 3, 2023 2:18:47 PM CET      January 3, 2023 2:18:47 PM CET      Success
Copying Flash Bios to Temp location      serveroda2                  January 3, 2023 2:18:47 PM CET      January 3, 2023 2:18:47 PM CET      Success
Starting the clusterware                 serveroda2                  January 3, 2023 2:20:44 PM CET      January 3, 2023 2:22:28 PM CET      Success
Registering image                        serveroda2                  January 3, 2023 2:22:28 PM CET      January 3, 2023 2:22:28 PM CET      Success
Registering working copy                 serveroda1                  January 3, 2023 2:22:28 PM CET      January 3, 2023 2:22:28 PM CET      Success
Registering image                        serveroda2                  January 3, 2023 2:22:28 PM CET      January 3, 2023 2:22:29 PM CET      Success
Creating GI home directories             serveroda2                  January 3, 2023 2:22:29 PM CET      January 3, 2023 2:22:29 PM CET      Success
Extract GI clone                         serveroda1                  January 3, 2023 2:22:29 PM CET      January 3, 2023 2:22:29 PM CET      Success
Extract GI clone                         serveroda2                  January 3, 2023 2:22:29 PM CET      January 3, 2023 2:22:29 PM CET      Success
Provisioning Software Only GI with RHP   serveroda1                  January 3, 2023 2:22:29 PM CET      January 3, 2023 2:22:29 PM CET      Success
Patch GI with RHP                        serveroda1                  January 3, 2023 2:22:29 PM CET      January 3, 2023 2:36:51 PM CET      Success
Updating GIHome version                  serveroda1                  January 3, 2023 2:36:52 PM CET      January 3, 2023 2:36:55 PM CET      Success
Updating GIHome version                  serveroda2                  January 3, 2023 2:36:52 PM CET      January 3, 2023 2:36:55 PM CET      Success
Validate GI availability                 serveroda1                  January 3, 2023 2:37:04 PM CET      January 3, 2023 2:37:05 PM CET      Success
Patch KVM CRS type                       serveroda1                  January 3, 2023 2:37:05 PM CET      January 3, 2023 2:37:05 PM CET      Success
Patch VM vDisks CRS dependencies         serveroda1                  January 3, 2023 2:37:05 PM CET      January 3, 2023 2:37:05 PM CET      Success
Patch DB System domain config            serveroda1                  January 3, 2023 2:37:05 PM CET      January 3, 2023 2:37:05 PM CET      Success
Update System version                    serveroda1                  January 3, 2023 2:37:05 PM CET      January 3, 2023 2:37:05 PM CET      Success
Update System version                    serveroda2                  January 3, 2023 2:37:05 PM CET      January 3, 2023 2:37:05 PM CET      Success
Cleanup JRE Home                         serveroda1                  January 3, 2023 2:37:06 PM CET      January 3, 2023 2:37:06 PM CET      Success
Add SYSNAME in Env                       serveroda2                  January 3, 2023 2:37:06 PM CET      January 3, 2023 2:37:06 PM CET      Success
Starting the clusterware                 serveroda2                  January 3, 2023 2:37:06 PM CET      January 3, 2023 2:37:06 PM CET      Success
Setting ACL for disk groups              serveroda1                  January 3, 2023 2:37:06 PM CET      January 3, 2023 2:37:13 PM CET      Success
Enable LKCE                              serveroda2                  January 3, 2023 2:39:09 PM CET      January 3, 2023 2:41:48 PM CET      Success
Enable LKCE                              serveroda1                  January 3, 2023 2:39:09 PM CET      January 3, 2023 2:41:47 PM CET      Success
Update previous workarounds              serveroda1                  January 3, 2023 2:41:49 PM CET      January 3, 2023 2:41:49 PM CET      Success
Update previous workarounds              serveroda2                  January 3, 2023 2:41:49 PM CET      January 3, 2023 2:41:49 PM CET      Success
Generating and saving BOM                serveroda2                  January 3, 2023 2:41:49 PM CET      January 3, 2023 2:43:17 PM CET      Success
Generating and saving BOM                serveroda1                  January 3, 2023 2:41:49 PM CET      January 3, 2023 2:43:22 PM CET      Success
PreRebootNode Actions                    serveroda1                  January 3, 2023 2:43:22 PM CET      January 3, 2023 2:44:18 PM CET      Success
PreRebootNode Actions                    serveroda2                  January 3, 2023 2:44:18 PM CET      January 3, 2023 2:45:11 PM CET      Success
Reboot Ilom                              serveroda1                  January 3, 2023 2:45:11 PM CET      January 3, 2023 2:45:11 PM CET      Success
Reboot Ilom                              serveroda2                  January 3, 2023 2:45:11 PM CET      January 3, 2023 2:45:11 PM CET      Success

Validate the components after the patch

[[email protected] ~]#  odacli describe-component
System Version
---------------
19.17.0.0.0

System node Name
---------------
serveroda1

Local System Version
---------------
19.17.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.17.0.0.0           up-to-date
GI
                                          19.17.0.0.221018      up-to-date
DB {
[OraDB19000_home1]
                                          19.10.0.0.210119      19.17.0.0.221018
[OraDB19000_home2]
                                          19.10.0.0.210119      19.17.0.0.221018
[OraDB19000_home3 [OMESTCT1,OMEMECD1,
OMECSCD1,OMECSCP1,OMESTCD1,OMEMECP1,
OMESTCP1,OMESTCT2]]                       19.14.0.0.220118      19.17.0.0.221018
}
DCSCONTROLLER
                                          19.17.0.0.0           up-to-date
DCSCLI
                                          19.17.0.0.0           up-to-date
DCSAGENT
                                          19.17.0.0.0           up-to-date
DCSADMIN
                                          19.17.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.1.0.23.r146986      up-to-date
BIOS
                                          41120100              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c3,c4]
                                          214.2.271.9           up-to-date
[c5]
                                          80000690              up-to-date
}
SHARED CONTROLLER FIRMWARE
                                          16.00.08.00           up-to-date
FIRMWAREEXPANDER
                                          0310                  up-to-date
LOCAL DISK FIRMWARE
                                          N2010112              up-to-date
SHARED DISK FIRMWARE {
[c0d0,c0d1]
                                          N2010112              up-to-date
[c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8]                           A170                  up-to-date
}
HMP
                                          2.4.8.9.603           up-to-date
System node Name
---------------
serveroda2

Local System Version
---------------
19.17.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.17.0.0.0           up-to-date
GI
                                          19.17.0.0.221018      up-to-date
DB {
[OraDB19000_home1]
                                          19.10.0.0.210119      19.17.0.0.221018
[OraDB19000_home2]
                                          19.10.0.0.210119      19.17.0.0.221018
[OraDB19000_home3 [OMESTCT1,OMEMECD1,
OMECSCD1,OMECSCP1,OMESTCD1,OMEMECP1,
OMESTCP1,OMESTCT2]]                       19.14.0.0.220118      19.17.0.0.221018
}
DCSCONTROLLER
                                          19.17.0.0.0           up-to-date
DCSCLI
                                          19.17.0.0.0           up-to-date
DCSAGENT
                                          19.17.0.0.0           up-to-date
DCSADMIN
                                          19.17.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.1.0.23.r146986      up-to-date
BIOS
                                          41120100              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c3]
                                          80000690              up-to-date
[c4,c5]
                                          214.2.271.9           up-to-date
}
SHARED CONTROLLER FIRMWARE
                                          16.00.08.00           up-to-date
FIRMWAREEXPANDER
                                          0310                  up-to-date
LOCAL DISK FIRMWARE
                                          N2010112              up-to-date
SHARED DISK FIRMWARE {
[c0d0,c0d1]
                                          N2010112              up-to-date
[c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
c1d7,c1d8,c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,
c2d6,c2d7,c2d8]                           A170                  up-to-date
}
HMP
                                          2.4.8.9.603           up-to-date

Update the storage components

Run a prepatch report for storage

[[email protected] ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -st -v 19.17.0.0.0

Job details
----------------------------------------------------------------
                     ID:  e50389ae-8944-44ff-b45a-c454cb33e050
            Description:  Patch pre-checks for [STORAGE]
                 Status:  Created
                Created:  January 3, 2023 2:57:54 PM CET
                Message:  Use 'odacli describe-prepatchreport -i e50389ae-8944-44ff-b45a-c454cb33e050' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
[[email protected] ~]# odacli describe-prepatchreport -i e50389ae-8944-44ff-b45a-c454cb33e050

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  e50389ae-8944-44ff-b45a-c454cb33e050
            Description:  Patch pre-checks for [STORAGE]
                 Status:  SUCCESS
                Created:  January 3, 2023 2:57:54 PM CET
                 Result:  All pre-checks succeeded
...
...

Patch the storage components

[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.17.0.0.0 --rolling
{
  "jobId" : "d02fa340-c82c-482e-9196-aac0d18d2dfc",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 15:02:10 PM CET",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "January 03, 2023 15:02:10 PM CET",
  "jobType" : null
}
[[email protected] ~]# odacli describe-job -i  "d02fa340-c82c-482e-9196-aac0d18d2dfc"

Job details
----------------------------------------------------------------
                     ID:  d02fa340-c82c-482e-9196-aac0d18d2dfc
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  January 3, 2023 3:02:10 PM CET
                Message:

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           serveroda1                  January 3, 2023 3:02:22 PM CET      January 3, 2023 3:02:30 PM CET      Success
Generating and saving BOM                serveroda1                  January 3, 2023 3:02:30 PM CET      January 3, 2023 3:04:15 PM CET      Success
Generating and saving BOM                serveroda2                  January 3, 2023 3:02:30 PM CET      January 3, 2023 3:04:10 PM CET      Success

Updating databases

For 19.17, we plan to update only some specific databases. In fact in 19.17 there is a command update-database

We first run the prepatch report for the corresponding dbhome

[[email protected] ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport --dbhome --dbhomeid 3d5f2d22-16db-4724-87e5-4965d6428e2e -v 19.17.0.0.0

Job details
----------------------------------------------------------------
                     ID:  cd643e18-1ed7-40db-81a9-cf03d8c1734e
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home3
                 Status:  Created
                Created:  January 3, 2023 3:07:49 PM CET
                Message:  Use 'odacli describe-prepatchreport -i cd643e18-1ed7-40db-81a9-cf03d8c1734e' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

We then validate that the report is fine

[[email protected] ~]# odacli describe-prepatchreport -i cd643e18-1ed7-40db-81a9-cf03d8c1734e

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  cd643e18-1ed7-40db-81a9-cf03d8c1734e
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home3
                 Status:  SUCCESS
                Created:  January 3, 2023 3:07:49 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
serveroda1

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          3d5f2d22-16db-4724-87e5-4965d6428e2e
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Node Name
---------------
serveroda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          3d5f2d22-16db-4724-87e5-4965d6428e2e
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

And we run the update-database command

[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-database -i a2b3ce19-1ce4-4394-b436-d0e4817d8111 -to ed8970e5-baf5-4b12-8209-809cbacbcbc7
{
  "jobId" : "97828641-18ee-4624-a150-c66b42dfae5f",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "January 03, 2023 15:28:40 PM CET",
  "resourceList" : [ ],
  "description" : "DB Patching: database ID is a2b3ce19-1ce4-4394-b436-d0e4817d8111",
  "updatedTime" : "January 03, 2023 15:28:40 PM CET",
  "jobType" : ""
}

A few minutes after the job fails

[[email protected] ~]# odacli describe-job -i "97828641-18ee-4624-a150-c66b42dfae5f"

Job details
----------------------------------------------------------------
                     ID:  97828641-18ee-4624-a150-c66b42dfae5f
            Description:  DB Patching: database ID is a2b3ce19-1ce4-4394-b436-d0e4817d8111
                 Status:  Failure
                Created:  January 3, 2023 3:28:40 PM CET
                Message:  DCS-10001:Internal error encountered: null.

Task Name                                Node Name                 Start Time                          End Time                            Status
---------------------------------------- ------------------------- ----------------------------------- ----------------------------------- ----------
DB Patching                              serveroda1                  January 3, 2023 3:28:43 PM CET      January 3, 2023 3:31:50 PM CET      Failure
DB Patching                              serveroda1                  January 3, 2023 3:28:43 PM CET      January 3, 2023 3:31:50 PM CET      Failure
task:TaskSequential_7670                 serveroda1                  January 3, 2023 3:28:50 PM CET      January 3, 2023 3:30:55 PM CET      Failure
Creating wallet for DB Client            serveroda1                  January 3, 2023 3:29:29 PM CET      January 3, 2023 3:29:30 PM CET      Success
Patch databases by RHP                   serveroda1                  January 3, 2023 3:29:30 PM CET      January 3, 2023 3:30:55 PM CET      Failure

We check the various but did not find the root cause. An SR is opened with Oracle

Conclusion

To resume patching an ODA HA is almost the same as patching a non-HA ODA server. We recommend to carefully read the documentation before starting the patch