In this blog, I describe the patching of an ODA X8 from 19.9 to 19.11. We use following patches

p32790643_1911000_Linux-x86-64.zip : Oracle Database Appliance Server Patch for ODACLI/DCS Stack
p30403673_1911000_Linux-x86-64.zip : Oracle Database Appliance GI Clone for ODACLI/DCS stack
p30403662_1911000_Linux-x86-64.zip : Oracle Database Appliance RDBMS Clone 19.11.0.0.210420 for ODACLI/DCS stack

Patching the Oracle Database Appliance Server

Before starting the process we first take a backup of the system with odabr ( ODA (Oracle Database Appliance): ODABR a System Backup/Restore Utility (Doc ID 2466177.1))

After dowloading the latest version, we do install odabr

[[email protected] software]# rpm -ivh odabr-2.0.1-70.noarch.rpm
warning: odabr-2.0.1-70.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID 939112d6: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:odabr-2.0.1-70                   ################################# [100%]

odabr-2.0.1.70 has been installed on /opt/odabr succesfully!

[[email protected] software]#

We can verify that there is actually no snapshots

[[email protected] software]# /opt/odabr/odabr infosnap

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.1-70
 Copyright 2013, 2021, Oracle and/or its affiliates.
 --------------------------------------------------------
 Author: Ruggero Citton <[email protected]>
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│


LVM snap name         Status                COW Size              Data%
-------------         ----------            ----------            ------
WARNING: 2022-08-30 09:07:58: LVM snapshot for 'root' does not exist
WARNING: 2022-08-30 09:07:58: LVM snapshot for 'opt' does not exist
WARNING: 2022-08-30 09:07:58: LVM snapshot for 'u01' does not exist

Let’s do a backup with odabr

[[email protected] software]# /opt/odabr/odabr backup -snap -rsize 15 -usize 90 -osize 60
INFO: 2022-08-30 09:08:51: Please check the logfile '/opt/odabr/out/log/odabr_35684.log' for more details


│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.1-70
 Copyright 2013, 2021, Oracle and/or its affiliates.
 --------------------------------------------------------
 Author: Ruggero Citton <[email protected]>
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

INFO: 2022-08-30 09:08:51: Checking superuser
INFO: 2022-08-30 09:08:51: Checking Bare Metal

INFO: 2022-08-30 09:08:51: Removing existing LVM snapshots
WARNING: 2022-08-30 09:08:51: LVM snapshot for 'opt' does not exist
WARNING: 2022-08-30 09:08:51: LVM snapshot for 'u01' does not exist
WARNING: 2022-08-30 09:08:51: LVM snapshot for 'root' does not exist

INFO: 2022-08-30 09:08:51: Checking current OS version
INFO: 2022-08-30 09:08:51: Checking LVM restore backgroud process
INFO: 2022-08-30 09:08:51: Checking LVM size

INFO: 2022-08-30 09:08:51: Boot device backup
INFO: 2022-08-30 09:08:51: Getting EFI device
INFO: 2022-08-30 09:08:51: ...step1 - unmounting EFI
INFO: 2022-08-30 09:08:51: ...step2 - making efi device backup
SUCCESS: 2022-08-30 09:08:52: ...EFI device backup saved as '/opt/odabr/out/hbi/efi.img'
INFO: 2022-08-30 09:08:52: ...step3 - checking EFI device backup
INFO: 2022-08-30 09:08:52: Getting boot device
INFO: 2022-08-30 09:08:52: ...step1 - making boot device backup using tar
SUCCESS: 2022-08-30 09:08:57: ...boot content saved as '/opt/odabr/out/hbi/boot.tar.gz'
INFO: 2022-08-30 09:08:57: ...step2 - unmounting boot
INFO: 2022-08-30 09:08:57: ...step3 - making boot device backup using dd
SUCCESS: 2022-08-30 09:08:59: ...boot device backup saved as '/opt/odabr/out/hbi/boot.img'
INFO: 2022-08-30 09:08:59: ...step4 - mounting boot
INFO: 2022-08-30 09:08:59: ...step5 - mounting EFI
INFO: 2022-08-30 09:08:59: ...step6 - checking boot device backup

INFO: 2022-08-30 09:08:59: Making OCR physical backup
INFO: 2022-08-30 09:09:02: ...ocr backup saved as '/opt/odabr/out/hbi/ocrbackup_35684.bck'
SUCCESS: 2022-08-30 09:09:02: OCR physical backup created successfully
INFO: 2022-08-30 09:09:02: OCR export backup
INFO: 2022-08-30 09:09:03: ...ocr export saved as '/opt/odabr/out/hbi/ocrexport_35684.bck'
SUCCESS: 2022-08-30 09:09:03: OCR export backup created successfully

INFO: 2022-08-30 09:09:03: Making LVM snapshot backup
SUCCESS: 2022-08-30 09:09:04: ...snapshot backup for 'opt' created successfully
SUCCESS: 2022-08-30 09:09:05: ...snapshot backup for 'u01' created successfully
SUCCESS: 2022-08-30 09:09:05: ...snapshot backup for 'root' created successfully
SUCCESS: 2022-08-30 09:09:05: LVM snapshots backup done successfully

We should now have snapshots

[[email protected] software]# /opt/odabr/odabr infosnap

│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.1-70
 Copyright 2013, 2021, Oracle and/or its affiliates.
 --------------------------------------------------------
 Author: Ruggero Citton <[email protected]>
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│


LVM snap name         Status                COW Size              Data%
-------------         ----------            ----------            ------
root_snap             active                15.00 GiB             0.01%
opt_snap              active                60.00 GiB             0.01%
u01_snap              active                90.00 GiB             0.01%


[[email protected] software]#

Now we can start the patching . First let’s unpack the patch p32790643_1911000_Linux-x86-64.zip

[[email protected] software]# unzip p32790643_1911000_Linux-x86-64.zip
Archive:  p32790643_1911000_Linux-x86-64.zip
 extracting: oda-sm-19.11.0.0.0-210530-server.zip
  inflating: README.txt
[[email protected] local_dumps]#

Then update the repository with the patch

[[email protected] software]# odacli describe-job -i "832fbb4d-2186-4b85-9978-7f642e04f3a9"

Job details
----------------------------------------------------------------
                     ID:  832fbb4d-2186-4b85-9978-7f642e04f3a9
            Description:  Repository Update
                 Status:  Success
                Created:  August 30, 2022 8:56:54 AM CEST
                Message:  /software/oda-sm-19.11.0.0.0-210530-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             August 30, 2022 8:56:54 AM CEST     August 30, 2022 8:57:18 AM CEST     Success

[[email protected] software]#

Validate that the job is successful

[[email protected] software]# odacli describe-job -i "832fbb4d-2186-4b85-9978-7f642e04f3a9"

Job details
----------------------------------------------------------------
                     ID:  832fbb4d-2186-4b85-9978-7f642e04f3a9
            Description:  Repository Update
                 Status:  Success
                Created:  August 30, 2022 8:56:54 AM CEST
                Message:  /software/oda-sm-19.11.0.0.0-210530-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             August 30, 2022 8:56:54 AM CEST     August 30, 2022 8:57:18 AM CEST     Success

[[email protected] software]#

Then we update the DCS_ADMIN

[[email protected] software]# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.11.0.0.0
{
  "jobId" : "fb14a1d0-52ce-43c3-871d-c93fdaff75b9",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 09:11:57 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAdmin patching",
  "updatedTime" : "August 30, 2022 09:11:57 AM CEST"
}

The job should return SUCCESS

[[email protected] software]# odacli describe-job -i "fb14a1d0-52ce-43c3-871d-c93fdaff75b9"

Job details
----------------------------------------------------------------
                     ID:  fb14a1d0-52ce-43c3-871d-c93fdaff75b9
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  August 30, 2022 9:11:57 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                August 30, 2022 9:11:57 AM CEST     August 30, 2022 9:11:57 AM CEST     Success
dcs-admin upgrade                        August 30, 2022 9:11:57 AM CEST     August 30, 2022 9:12:03 AM CEST     Success

[[email protected] software]#

Let’s update the DCS components

[[email protected] software]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.11.0.0.                        0
{
  "jobId" : "a274b632-92a8-4f8f-b05d-b3c5d7c67dae",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL installation is successful. Metadata migration is successful. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  DCS-Agent restart is successful. ",
  "reports" : null,
  "createTimestamp" : "August 30, 2022 09:13:37 AM CEST",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "August 30, 2022 09:15:41 AM CEST"
}

[[email protected] software]#

The describe-job for this job doesn’t return anything. It’s normal, don’t worry

[[email protected] software]# odacli describe-job -i "a274b632-92a8-4f8f-b05d-b3c5d7c67dae"
DCS-10000:Resource Job with ID a274b632-92a8-4f8f-b05d-b3c5d7c67dae is not found.

We can see the status of our components at this point

[[email protected] software]# odacli describe-component
System Version
---------------
19.11.0.0.0

System node Name
---------------
oda-prod

Local System Version
---------------
19.11.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.9.0.0.0            19.11.0.0.0

GI                                        19.9.0.0.201020       19.11.0.0.210420

DB {
[ OraDB19000_home1 ]                      19.9.0.0.201020       19.11.0.0.210420
[ OraDB11204_home1 ]                      11.2.0.4.201020       not-available
}

DCSAGENT                                  19.11.0.0.0           up-to-date

OS                                        7.8                   7.9

ILOM                                      5.0.1.21.r136383      5.0.1.21.a.r138015

BIOS                                      52030400              up-to-date

FIRMWARECONTROLLER                        VDV1RL04              up-to-date

FIRMWAREDISK                              1102                  up-to-date

ASR                                       20.3.0                up-to-date

HMP                                       2.4.7.0.1             2.4.8.0.600


[[email protected] software]#

We can now update the DCS agent

[[email protected] software]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.11.0.0.0
{
  "jobId" : "0ea4fc54-4797-47a0-966d-de1f3d14bda4",
  "status" : "Created",
  "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands",
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 09:21:11 AM CEST",
  "resourceList" : [ ],
  "description" : "DcsAgent patching",
  "updatedTime" : "August 30, 2022 09:21:11 AM CEST"
}
[[email protected] software]# 

The job status should return SUCCESS

[[email protected] software]# odacli describe-job -i "0ea4fc54-4797-47a0-966d-de1f3d14bda4"

Job details
----------------------------------------------------------------
                     ID:  0ea4fc54-4797-47a0-966d-de1f3d14bda4
            Description:  DcsAgent patching
                 Status:  Success
                Created:  August 30, 2022 9:21:11 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
dcs-agent upgrade  to version 19.11.0.0.0 August 30, 2022 9:21:12 AM CEST     August 30, 2022 9:23:32 AM CEST     Success
Update System version                    August 30, 2022 9:23:32 AM CEST     August 30, 2022 9:23:32 AM CEST     Success

[[email protected] software]#

Now it’s time to update the repository with grid patch

We unpack the corresponding patch

[[email protected] software]# unzip p30403673_1911000_Linux-x86-64.zip
Archive:  p30403673_1911000_Linux-x86-64.zip
 extracting: odacli-dcs-19.11.0.0.0-210530-GI-19.11.0.0.zip
replace README.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
  inflating: README.txt
[[email protected]a-prod software]# ls -l odacli-dcs-19.11.0.0.0-210530-GI-19.11.0.0.zip
-rwxrwxrwx 1 root root 5550326480 May 31  2021 odacli-dcs-19.11.0.0.0-210530-GI-19.11.0.0.zip
[[email protected] software]#

We update the repository with the patch

[[email protected] software]# /opt/oracle/dcs/bin/odacli update-repository -f /software/odacli-dcs-19.11.0.0.0-210530-GI-19.11.0.0.zip
{
  "jobId" : "17adc146-c10f-418a-b08b-f7d4c9390cbc",
  "status" : "Created",
  "message" : "/software/odacli-dcs-19.11.0.0.0-210530-GI-19.11.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 09:27:17 AM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "August 30, 2022 09:27:17 AM CEST"
}
[[email protected] software]#

Be sure that the job was fine

[[email protected] software]# odacli describe-job -i "17adc146-c10f-418a-b08b-f7d4c9390cbc"

Job details
----------------------------------------------------------------
                     ID:  17adc146-c10f-418a-b08b-f7d4c9390cbc
            Description:  Repository Update
                 Status:  Success
                Created:  August 30, 2022 9:27:17 AM CEST
                Message:  /software/odacli-dcs-19.11.0.0.0-210530-GI-19.11.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             August 30, 2022 9:27:20 AM CEST     August 30, 2022 9:27:58 AM CEST     Success
registering image                        August 30, 2022 9:27:58 AM CEST     August 30, 2022 9:27:58 AM CEST     Success

[[email protected] software]#

Before applying the patch, we can run the prepatch checks

[[email protected] software]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.11.0.0.0

Job details
----------------------------------------------------------------
                     ID:  09ff843d-1b4c-4a90-91ec-61004684903e
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  Created
                Created:  August 30, 2022 9:30:58 AM CEST
                Message:  Use 'odacli describe-prepatchreport -i 09ff843d-1b4c-4a90-91ec-61004684903e' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] software]#

The results can be found below

[[email protected] software]# odacli describe-prepatchreport -i 09ff843d-1b4c-4a90-91ec-61004684903e

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  09ff843d-1b4c-4a90-91ec-61004684903e
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  August 30, 2022 9:30:58 AM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
oda-prod

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.11.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.11.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.11.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate clones location exist  Success   Validated clones location
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution



[[email protected] software]#

As everything is fine, we can apply the patch. We just ensure that we have at least 20G of free space for / /opt and /u01

[[email protected] software]# df -h / /opt /u01
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot   30G  6.5G   22G  24% /
/dev/mapper/VolGroupSys-LogVolOpt    99G   33G   61G  36% /opt
/dev/mapper/VolGroupSys-LogVolU01    99G   65G   29G  70% /u01
[[email protected] software]#

And we apply the patch for the server

[[email protected] software]# /opt/oracle/dcs/bin/odacli update-server -v 19.11.0.0.0
{
  "jobId" : "5d3a7493-4426-4272-8ffc-59fad66a1eab",
  "status" : "Created",
  "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 09:37:06 AM CEST",
  "resourceList" : [ ],
  "description" : "Server Patching",
  "updatedTime" : "August 30, 2022 09:37:06 AM CEST"
}
[[email protected] software]#

A few moments later we verify that the patch was fine

[[email protected] log]# odacli describe-job -i 5d3a7493-4426-4272-8ffc-59fad66a1eab

Job details
----------------------------------------------------------------
                     ID:  5d3a7493-4426-4272-8ffc-59fad66a1eab
            Description:  Server Patching
                 Status:  Success
                Created:  August 30, 2022 9:37:06 AM CEST
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                August 30, 2022 9:37:14 AM CEST     August 30, 2022 9:37:14 AM CEST     Success
dcs-controller upgrade                   August 30, 2022 9:37:17 AM CEST     August 30, 2022 9:37:17 AM CEST     Success
Creating repositories using yum          August 30, 2022 9:37:17 AM CEST     August 30, 2022 9:37:19 AM CEST     Success
Updating YumPluginVersionLock rpm        August 30, 2022 9:37:19 AM CEST     August 30, 2022 9:37:19 AM CEST     Success
Applying OS Patches                      August 30, 2022 9:37:19 AM CEST     August 30, 2022 9:46:49 AM CEST     Success
Creating repositories using yum          August 30, 2022 9:46:49 AM CEST     August 30, 2022 9:46:49 AM CEST     Success
Applying HMP Patches                     August 30, 2022 9:46:49 AM CEST     August 30, 2022 9:47:05 AM CEST     Success
Patch location validation                August 30, 2022 9:47:05 AM CEST     August 30, 2022 9:47:05 AM CEST     Success
oda-hw-mgmt upgrade                      August 30, 2022 9:47:05 AM CEST     August 30, 2022 9:47:33 AM CEST     Success
OSS Patching                             August 30, 2022 9:47:33 AM CEST     August 30, 2022 9:47:33 AM CEST     Success
Applying Firmware Disk Patches           August 30, 2022 9:47:34 AM CEST     August 30, 2022 9:47:40 AM CEST     Success
Applying Firmware Controller Patches     August 30, 2022 9:47:40 AM CEST     August 30, 2022 9:47:44 AM CEST     Success
Checking Ilom patch Version              August 30, 2022 9:47:44 AM CEST     August 30, 2022 9:47:45 AM CEST     Success
Patch location validation                August 30, 2022 9:47:45 AM CEST     August 30, 2022 9:47:45 AM CEST     Success
Save password in Wallet                  August 30, 2022 9:47:45 AM CEST     August 30, 2022 9:47:45 AM CEST     Success
Apply Ilom patch                         August 30, 2022 9:47:45 AM CEST     August 30, 2022 9:56:18 AM CEST     Success
Copying Flash Bios to Temp location      August 30, 2022 9:56:18 AM CEST     August 30, 2022 9:56:18 AM CEST     Success
Patch location validation                August 30, 2022 9:56:18 AM CEST     August 30, 2022 9:56:18 AM CEST     Success
ASR Manager RPM update                   August 30, 2022 9:56:19 AM CEST     August 30, 2022 9:56:19 AM CEST     Success
Modify JavaExec Path                     August 30, 2022 9:56:19 AM CEST     August 30, 2022 9:56:19 AM CEST     Success
Remove AsrConfBackup File                August 30, 2022 9:56:19 AM CEST     August 30, 2022 9:56:19 AM CEST     Success
Starting the clusterware                 August 30, 2022 9:56:19 AM CEST     August 30, 2022 9:58:23 AM CEST     Success
create ASM priv network                  August 30, 2022 9:58:23 AM CEST     August 30, 2022 9:58:27 AM CEST     Success
configure ASM priv network               August 30, 2022 9:58:27 AM CEST     August 30, 2022 9:58:29 AM CEST     Success
Starting the clusterware                 August 30, 2022 9:58:29 AM CEST     August 30, 2022 9:58:29 AM CEST     Success
registering image                        August 30, 2022 9:58:29 AM CEST     August 30, 2022 9:58:29 AM CEST     Success
registering working copy                 August 30, 2022 9:58:29 AM CEST     August 30, 2022 9:58:30 AM CEST     Success
registering image                        August 30, 2022 9:58:30 AM CEST     August 30, 2022 9:58:30 AM CEST     Success
Creating GI home directories             August 30, 2022 9:58:30 AM CEST     August 30, 2022 9:58:30 AM CEST     Success
Extract Gi clone                         August 30, 2022 9:58:30 AM CEST     August 30, 2022 9:59:55 AM CEST     Success
Provisioning Software Only GI with RHP   August 30, 2022 9:59:55 AM CEST     August 30, 2022 10:00:56 AM CEST    Success
Patch GI with RHP                        August 30, 2022 10:00:56 AM CEST    August 30, 2022 10:09:13 AM CEST    Success
Updating GiHome version                  August 30, 2022 10:09:13 AM CEST    August 30, 2022 10:09:15 AM CEST    Success
Starting the clusterware                 August 30, 2022 10:09:19 AM CEST    August 30, 2022 10:09:19 AM CEST    Success
remove network public interface          August 30, 2022 10:09:19 AM CEST    August 30, 2022 10:09:22 AM CEST    Success
create bridge network                    August 30, 2022 10:09:22 AM CEST    August 30, 2022 10:09:27 AM CEST    Success
modify network public interface          August 30, 2022 10:09:27 AM CEST    August 30, 2022 10:09:28 AM CEST    Success
Update System version                    August 30, 2022 10:09:28 AM CEST    August 30, 2022 10:09:28 AM CEST    Success
Cleanup JRE Home                         August 30, 2022 10:09:28 AM CEST    August 30, 2022 10:09:28 AM CEST    Success
Add SYSNAME in Env                       August 30, 2022 10:09:28 AM CEST    August 30, 2022 10:09:28 AM CEST    Success
Setting ACL for disk groups              August 30, 2022 10:09:28 AM CEST    August 30, 2022 10:09:31 AM CEST    Success
preRebootNode Actions                    August 30, 2022 10:10:39 AM CEST    August 30, 2022 10:11:30 AM CEST    Success
Reboot Ilom                              August 30, 2022 10:11:30 AM CEST    August 30, 2022 10:11:30 AM CEST    Success

[[email protected] log]#

Below the component version. We can see that all components are up-to-date except the Oracle databases 19c. We will path the database later

[[email protected] ~]# odacli describe-component
System Version
---------------
19.11.0.0.0

System node Name
---------------
oda-prod

Local System Version
---------------
19.11.0.0.0

Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.11.0.0.0           up-to-date

GI                                        19.11.0.0.210420      up-to-date

DB {
[ OraDB19000_home1 ]                      19.9.0.0.201020       19.11.0.0.210420
[ OraDB11204_home1 ]                      11.2.0.4.201020       not-available
}

DCSAGENT                                  19.11.0.0.0           up-to-date

OS                                        7.9                   up-to-date

ILOM                                      5.0.1.21.a.r138015    up-to-date

BIOS                                      52030400              up-to-date

FIRMWARECONTROLLER                        VDV1RL04              up-to-date

FIRMWAREDISK                              1102                  up-to-date

ASR                                       20.3.0                up-to-date

HMP                                       2.4.8.0.600           up-to-date


[[email protected] ~]#

To patch the storage, we run the following command

[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-storage -v 19.11.0.0.0
{
  "jobId" : "eea8b12b-76d1-4695-9b29-f0102a8a67da",
  "status" : "Created",
  "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 10:16:38 AM CEST",
  "resourceList" : [ ],
  "description" : "Storage Firmware Patching",
  "updatedTime" : "August 30, 2022 10:16:38 AM CEST"
}
[[email protected] ~]#

The Job returns SUCCESS

[[email protected] ~]# odacli describe-job -i "eea8b12b-76d1-4695-9b29-f0102a8a67da"

Job details
----------------------------------------------------------------
                     ID:  eea8b12b-76d1-4695-9b29-f0102a8a67da
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  August 30, 2022 10:16:38 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           August 30, 2022 10:16:45 AM CEST    August 30, 2022 10:16:49 AM CEST    Success
preRebootNode Actions                    August 30, 2022 10:16:50 AM CEST    August 30, 2022 10:16:50 AM CEST    Success
Reboot Ilom                              August 30, 2022 10:16:50 AM CEST    August 30, 2022 10:16:50 AM CEST    Success

[[email protected] ~]#

As the patch went fine, we can remove the odabr snapshots now

[[email protected] software]# /opt/odabr/odabr  delsnap
INFO: 2022-08-30 10:22:10: Please check the logfile '/opt/odabr/out/log/odabr_64634.log' for more details

INFO: 2022-08-30 10:22:10: Removing LVM snapshots
INFO: 2022-08-30 10:22:10: ...removing LVM snapshot for 'opt'
SUCCESS: 2022-08-30 10:22:11: ...snapshot for 'opt' removed successfully
INFO: 2022-08-30 10:22:11: ...removing LVM snapshot for 'u01'
SUCCESS: 2022-08-30 10:22:11: ...snapshot for 'u01' removed successfully
INFO: 2022-08-30 10:22:11: ...removing LVM snapshot for 'root'
SUCCESS: 2022-08-30 10:22:11: ...snapshot for 'root' removed successfully
SUCCESS: 2022-08-30 10:22:11: Remove LVM snapshots done successfully
[[email protected] software]# 

Patching existing databases

In my case only 19c databases are updated. Oracle 11g is not supported for the update.

The first step is to unpack the corresponding file

[[email protected] software]# unzip p30403662_1911000_Linux-x86-64.zip
Archive:  p30403662_1911000_Linux-x86-64.zip
 extracting: odacli-dcs-19.11.0.0.0-210530-DB-19.11.0.0.zip
  inflating: README.txt
[[email protected] software]#
[[email protected] software]# ls -l odacli-dcs-19.11.0.0.0-210530-DB-19.11.0.0.zip
-rwxrwxrwx 1 root root 4953108233 May 31  2021 odacli-dcs-19.11.0.0.0-210530-DB-19.11.0.0.zip
[[email protected] software]#

We load the DB 19c clone on the repository

[[email protected] software]# /opt/oracle/dcs/bin/odacli update-repository -f /software/odacli-dcs-19.11.0.0.0-210530-DB-19.11.0.0.zip
{
  "jobId" : "87445cac-e643-474c-87ec-93b7811caa17",
  "status" : "Created",
  "message" : "/software/odacli-dcs-19.11.0.0.0-210530-DB-19.11.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 10:47:07 AM CEST",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "August 30, 2022 10:47:07 AM CEST"
}
[[email protected] software]#

We validate the status of the job

[[email protected] software]# odacli describe-job -i "87445cac-e643-474c-87ec-93b7811caa17"

Job details
----------------------------------------------------------------
                     ID:  87445cac-e643-474c-87ec-93b7811caa17
            Description:  Repository Update
                 Status:  Success
                Created:  August 30, 2022 10:47:07 AM CEST
                Message:  /software/odacli-dcs-19.11.0.0.0-210530-DB-19.11.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             August 30, 2022 10:47:09 AM CEST    August 30, 2022 10:47:44 AM CEST    Success
registering image                        August 30, 2022 10:47:44 AM CEST    August 30, 2022 10:47:45 AM CEST    Success

[[email protected] software]#

In the documentation we can fine following lines

Starting with Oracle Database Appliance release 19.11, Oracle Database homes are not created on the local disk. Oracle Database homes are created on an Oracle ACFS-managed file system. For a database user oracle, the new database homes are created under /u01/app/odaorahome/oracle/.

 Before patching the database home, configure the database home storage with the odacli configure-dbhome-storage command:

Then we create the storage for the Dbhome

[[email protected] software]# odacli configure-dbhome-storage -dg DATA
{
  "jobId" : "c82c5d33-fe67-4092-86d8-5746db2ac447",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 10:50:07 AM CEST",
  "resourceList" : [ ],
  "description" : "Configure database home storage",
  "updatedTime" : "August 30, 2022 10:50:07 AM CEST"
}
You have new mail in /var/spool/mail/root
[[email protected] software]#

The status was fine

[[email protected] software]# odacli describe-job -i "c82c5d33-fe67-4092-86d8-5746db2ac447"

Job details
----------------------------------------------------------------
                     ID:  c82c5d33-fe67-4092-86d8-5746db2ac447
            Description:  Configure database home storage
                 Status:  Success
                Created:  August 30, 2022 10:50:07 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Configuring database home storage        August 30, 2022 10:50:07 AM CEST    August 30, 2022 10:50:07 AM CEST    Success

[[email protected] software]#



run the prepatch
[[email protected] software]# odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
2a82f095-5952-42fa-b22c-a81ca6da5f30     OraDB19000_home1     19.9.0.0.201020                          /u01/app/oracle/product/19.0.0.0/dbhome_1     CONFIGURED
100c93b2-0f50-43e6-a04f-bb4a78c41aeb     OraDB11204_home1     11.2.0.4.201020                          /u01/app/oracle/product/11.2.0.4/dbhome_1     CONFIGURED

[[email protected] software]#

Before patching the HOMES, we do a precheck.

[[email protected] software]# /opt/oracle/dcs/bin/odacli create-prepatchreport --dbhome --dbhomeid 2a82f095-5952-42fa-b22c-a81ca6da5f30 -v 19.11.0.0.0

Job details
----------------------------------------------------------------
                     ID:  8ac2998e-ce0e-47e5-b9c3-6e5fd0c49139
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  Created
                Created:  August 30, 2022 10:52:56 AM CEST
                Message:  Use 'odacli describe-prepatchreport -i 8ac2998e-ce0e-47e5-b9c3-6e5fd0c49139' to check details of results

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

[[email protected] software]#

Below the results

[[email protected] software]# odacli describe-prepatchreport -i 8ac2998e-ce0e-47e5-b9c3-6e5fd0c49139

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  8ac2998e-ce0e-47e5-b9c3-6e5fd0c49139
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  FAILED
                Created:  August 30, 2022 10:52:56 AM CEST
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
oda-prod

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          2a82f095-5952-42fa-b22c-a81ca6da5f30
Validate patching tag           Success   Validated patching tag: 19.11.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Is DB clone registered          Success   Successfully validated clone
                                          registered
Is DB clone available           Success   Successfully validated clone file
                                          exists
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Verify the Alternate Archive    Failed    AHF-4940: One or more log archive
Destination is Configured to              destination and alternate log archive
Prevent Database Hangs                    destination settings are not as
                                          recommended
Verify the Alternate Archive    Failed    AHF-4940: One or more log archive
Destination is Configured to              destination and alternate log archive
Prevent Database Hangs                    destination settings are not as
                                          recommended
Verify the Alternate Archive    Failed    AHF-4940: One or more log archive
Destination is Configured to              destination and alternate log archive
Prevent Database Hangs                    destination settings are not as
                                          recommended
Verify the Alternate Archive    Failed    AHF-4940: One or more log archive
Destination is Configured to              destination and alternate log archive
Prevent Database Hangs                    destination settings are not as
                                          recommended
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days



[[email protected] software]#

We got some oracheck errors. After a look on Oracle documentation of the known Issues we found following lines

AHF error in prepatch report for the update-dbhome command

When you patch server to Oracle Database Appliance release 19.13, the odacli update-dbhome command may fail.

The following error message is displayed in the pre-patch report:

Verify the Alternate Archive    Failed    AHF-4940: One or more log archive

Destination is Configured to              destination and alternate log archive

Prevent Database Hangs                    destination settings are not as recommended          

Hardware Models

All Oracle Database Appliance hardware models

Workaround

Follow these steps:

    Run the odacli update-dbhome command with the -f option.

    /opt/oracle/dcs/bin/odacli update-dbhome –dbhomeid 7c67c5b4-f585-4ba9-865f-c719c63c0a6e -v 19.13.0.0.0 -f

This issue is tracked with Oracle bug 33144170.

For the error relative to the global_name, we change the value to true and will set it back to false after the patch

And then we run the patch with the option -f

[[email protected] archivelog]# /opt/oracle/dcs/bin/odacli update-dbhome --dbhomeid 2a82f095-5952-42fa-b22c-a81ca6da5f30 -v 19.11.0.0.0 -f

{
  "jobId" : "4b499623-9ca9-4089-b18b-5c13af1d9067",
  "status" : "Created",
  "message" : null,
  "reports" : [ ],
  "createTimestamp" : "August 30, 2022 11:15:58 AM CEST",
  "resourceList" : [ ],
  "description" : "DB Home Patching: Home Id is 2a82f095-5952-42fa-b22c-a81ca6da5f30",
  "updatedTime" : "August 30, 2022 11:15:58 AM CEST"
}
[[email protected] archivelog]#

A few moments after the patch was fine

[[email protected] ~]# odacli describe-job -i "4b499623-9ca9-4089-b18b-5c13af1d9067"

Job details
----------------------------------------------------------------
                     ID:  4b499623-9ca9-4089-b18b-5c13af1d9067
            Description:  DB Home Patching: Home Id is 2a82f095-5952-42fa-b22c-a81ca6da5f30
                 Status:  Success
                Created:  August 30, 2022 11:15:58 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating ACFS database home              August 30, 2022 11:16:11 AM CEST    August 30, 2022 11:17:01 AM CEST    Success
Validating dbHome available space        August 30, 2022 11:17:01 AM CEST    August 30, 2022 11:17:01 AM CEST    Success
Creating DbHome Directory                August 30, 2022 11:17:01 AM CEST    August 30, 2022 11:17:01 AM CEST    Success
Create required directories              August 30, 2022 11:17:01 AM CEST    August 30, 2022 11:17:01 AM CEST    Success
Extract DB clones                        August 30, 2022 11:17:01 AM CEST    August 30, 2022 11:18:09 AM CEST    Success
ProvDbHome by using RHP                  August 30, 2022 11:18:09 AM CEST    August 30, 2022 11:19:11 AM CEST    Success
Enable DB options                        August 30, 2022 11:19:11 AM CEST    August 30, 2022 11:19:26 AM CEST    Success
Creating wallet for DB Client            August 30, 2022 11:19:29 AM CEST    August 30, 2022 11:19:29 AM CEST    Success
Creating wallet for DB Client            August 30, 2022 11:19:53 AM CEST    August 30, 2022 11:19:53 AM CEST    Success
Patch databases by RHP                   August 30, 2022 11:19:53 AM CEST    August 30, 2022 11:24:11 AM CEST    Success
updating database metadata               August 30, 2022 11:24:54 AM CEST    August 30, 2022 11:24:54 AM CEST    Success
Patch databases by RHP                   August 30, 2022 11:24:54 AM CEST    August 30, 2022 11:29:01 AM CEST    Success
updating database metadata               August 30, 2022 11:29:41 AM CEST    August 30, 2022 11:29:41 AM CEST    Success
Update System version                    August 30, 2022 11:29:41 AM CEST    August 30, 2022 11:29:41 AM CEST    Success
Updating metadata for database           August 30, 2022 11:30:09 AM CEST    August 30, 2022 11:30:09 AM CEST    Success
Updating metadata for database           August 30, 2022 11:30:09 AM CEST    August 30, 2022 11:30:09 AM CEST    Success

[[email protected] ~]#

Connecting to the 19c database we can verify that the RU is now 19.11

SQL> select description from dba_registry_sqlpatch;

DESCRIPTION
--------------------------------------------------------------------------------
OJVM RELEASE UPDATE: 19.9.0.0.201020 (31668882)
Database Release Update : 19.9.0.0.201020 (31771877)
OJVM RELEASE UPDATE: 19.9.0.0.201020 (31668882)
OJVM RELEASE UPDATE: 19.11.0.0.210420 (32399816)
Database Release Update : 19.11.0.0.210420 (32545013)

SQL>

Conclusion

Hope this blog help