Introduction

Patch 19.15 is now available on Oracle Database Appliance. It’s time to test it.

What’s new?

This version brings April’s PSU to database and grid homes with their bug fixes, as usual. It also brings latest database 21.6 but only for DB Systems (21c being an innovation release). As you may know, new ODA X9-2 is here with 3 models: X9-2S, X9-2L and X9-2HA. This release is the first to support these brand new appliances.

Most important feature is the new “Data Preserving Reprovisioning”: you can now reimage your ODA without erasing the DATA disks. It could be the definitive solution for clean patching without a complete reimaging.

Which ODA is compatible with this 19.15?

The brand new ODAs X9-2S/L/HA (basically a refresh of previous X8-2S/M/HA) is for sure supported, so are the X8, X7 and X6 series. X5-2HA is still on the compatibility list, but most of them are at least 6 years old now.

Is this patch a cumulative one?

This 19.15 can be applied on top of 19.11 or later. If you’re using older versions, you may think about using the new “Data Preserving Reprovisioning”, let’s call it DPR, the promise being exciting. But this is not for everyone: this DPR feature is limited to these releases: 12.1.2.12, 12.2.1.4, 18.3, 18.5, 18.7, and 18.8. If you don’t use these releases, you will first need to jump to supported next one. This DPR feature is also not available if you already use one of the 19.x version.

Is there also a patch for my databases?

Only databases version 12.1 and 19c are now supported. Classic patching without DPR will preserve your existing binaries for unsupported versions.

Download the patch and clone files

Download the patch and the corresponding clones to be able to apply the complete patch.

  • 34069644 => the patch itself
  • 30403673 => the GI clone needed for deploying new version (mandatory)
  • 30403662 => the DB clone for deploying new version of 19c (if you use 19c databases)
  • 23494992 => the DB clone for deploying new version of 12.1 (if you use 12.1 databases)

Be sure to choose the very latest 19.15 when downloading the clones, download link will first propose older versions.

In this demo, I will not be able to use DPR because my ODA is already running 19.14.

Prepare the patching

Before running pre-patch, please check these prerequisites:

  • filesystems have 20% available free space (does not concern acfs volumes)
  • additional rpms manually installed should be removed
  • revert profile scripts to default’s one (for grid and oracle users)
  • make sure you can afford longer than planned downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much.

You can use odabr to backup your filesystems to snapshot or to nfs, or simply backup all your important files to a nfs share in case patching would fail.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB {
[ OraDB19000_home1 ]
                                          19.12.0.0.210720      19.14.0.0.220118
[ OraDB19000_home4,OraDB19000_home6,
OraDB19000_home7 ]                        19.14.0.0.220118      up-to-date
}
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be updated with versions provided within the patch.

Patching from 19.14 will normally be easy.

Prepararing the patch and updating the DCS tools

Copy the patch files on your ODA in a temp directory. Then unzip the files:

cd /opt/dbi/
for f in p*1915000*.zip; do unzip -n $f; done
Archive:  p30403662_1915000_Linux-x86-64.zip
 extracting: odacli-dcs-19.15.0.0.0-220425-DB-19.15.0.0.zip
  inflating: README.txt
Archive:  p30403673_1915000_Linux-x86-64.zip
 extracting: odacli-dcs-19.15.0.0.0-220425-GI-19.15.0.0.zip
Archive:  p34069644_1915000_Linux-x86-64.zip
 extracting: oda-sm-19.15.0.0.0-220530-server.zip

rm -rf p*1915000*.zip

Register the patch in the repository:

odacli update-repository -f /opt/dbi/oda-sm-19.15.0.0.0-220530-server.zip

odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           19.15.0.0.0
GI
                                          19.14.0.0.220118      19.15.0.0.220419
DB {
[ OraDB19000_home1 ]
                                          19.12.0.0.210720      19.15.0.0.220419
[ OraDB19000_home4,OraDB19000_home6,
OraDB19000_home7 ]                        19.14.0.0.220118      19.15.0.0.220419
}
DCSCONTROLLER
                                          19.14.0.0.0           19.15.0.0.0
DCSCLI
                                          19.14.0.0.0           19.15.0.0.0
DCSAGENT
                                          19.14.0.0.0           19.15.0.0.0
DCSADMIN
                                          19.14.0.0.0           19.15.0.0.0
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           2.4.8.0.601

Update the DCS tooling of your ODA:

/opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.15.0.0.0
sleep 60;  /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.15.0.0.0
/opt/oracle/dcs/bin/odacli update-dcsagent -v 19.15.0.0.0

Note that updating the DCS components is not done through a job:

sleep 180; odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 4
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
8b25f005-0b51-4fc3-bfbc-fc7fe28a2219     Repository Update                                                           June 9, 2022 8:14:53 AM CEST        Success
be22387a-e0e1-4c3e-9cd0-211700ba0679     DcsAdmin patching                                                           June 9, 2022 8:16:19 AM CEST        Success
7a995e7c-b2ba-4b76-b559-6a683062800a     DcsAgent patching                                                           June 9, 2022 8:19:26 AM CEST        Success

Now you can register GI and DB clones:

odacli update-repository -f /opt/dbi/odacli-dcs-19.15.0.0.0-220425-GI-19.15.0.0.zip
sleep 50; odacli update-repository -f /opt/dbi/odacli-dcs-19.15.0.0.0-220425-DB-19.15.0.0.zip

sleep 50; odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 3
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
51e165e6-eb8a-4421-885a-09aeacb91613     Repository Update                                                           June 9, 2022 8:25:32 AM CEST        Success
d0db4cf9-0f2a-439e-a219-0b30dbec76f3     Repository Update                                                           June 9, 2022 8:26:37 AM CEST        Success

odacli describe-component | grep -v ^$
System Version
---------------
19.15.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.15.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           19.15.0.0.0
GI
                                          19.14.0.0.220118      19.15.0.0.220419
DB {
[OraDB19000_home1]
                                          19.12.0.0.210720      19.15.0.0.220419
[OraDB19000_home4 [wra]]
                                          19.14.0.0.220118      19.15.0.0.220419
[OraDB19000_home6 [roman]]
                                          19.14.0.0.220118      19.15.0.0.220419
[OraDB19000_home7 [DHE,MAW,TRSNOC,
TSYCDB1,TSYCDB2]]                         19.14.0.0.220118      19.15.0.0.220419
}
DCSCONTROLLER
                                          19.15.0.0.0           up-to-date
DCSCLI
                                          19.15.0.0.0           up-to-date
DCSAGENT
                                          19.15.0.0.0           up-to-date
DCSADMIN
                                          19.15.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c3]
[c4]
80007BC7 8000A87E 
} 
SHARED CONTROLLER FIRMWARE 
                                         VDV1RL04            up-to-date 
LOCAL DISK FIRMWARE           1132             up-to-date 
SHARED DISK FIRMWARE        1132             up-to-date 
HMP                               2.4.8.0.600             2.4.8.0.601

This update will be for Oracle software and also for some microcodes on my X8-2M.

Pre-patching report

Let’s do the pre-patching test:

odacli create-prepatchreport -s -v 19.15.0.0.0

sleep 500; odacli describe-prepatchreport -i 508808ef-dddd-4121-b7e8-1ba717c895ba

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  508808ef-dddd-4121-b7e8-1ba717c895ba
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  SUCCESS
                Created:  June 9, 2022 8:42:08 AM CEST
                 Result:  All pre-checks succeeded

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.15.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.15.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Patch already applied
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.15.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

On my configuration it didn’t work on the first try because of an old 19.12 GI. I removed it manually:

rm -fr /u01/app/19.12.0.0/grid/

Everything is OK to start patching.

Patching infrastructure and GI

Let’s start the update-server:

odacli update-server -v 19.15.0.0.0
odacli describe-job -i 6ad4a7f1-cce9-42c0-929e-e509383311b4

Job details
----------------------------------------------------------------
                     ID:  6ad4a7f1-cce9-42c0-929e-e509383311b4
            Description:  Server Patching
                 Status:  Success
                Created:  June 9, 2022 8:51:32 AM CEST
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              June 9, 2022 8:51:47 AM CEST        June 9, 2022 8:51:47 AM CEST        Success
Validate ILOM server reachable           June 9, 2022 8:51:47 AM CEST        June 9, 2022 8:51:47 AM CEST        Success
Validate DCS Admin mTLS setup            June 9, 2022 8:51:47 AM CEST        June 9, 2022 8:51:47 AM CEST        Success
Configure export clones resource         June 9, 2022 8:51:48 AM CEST        June 9, 2022 8:51:48 AM CEST        Success
Creating repositories using yum          June 9, 2022 8:51:48 AM CEST        June 9, 2022 8:51:51 AM CEST        Success
Updating YumPluginVersionLock rpm        June 9, 2022 8:51:51 AM CEST        June 9, 2022 8:51:51 AM CEST        Success
Applying OS Patches                      June 9, 2022 8:51:51 AM CEST        June 9, 2022 8:59:00 AM CEST        Success
Creating repositories using yum          June 9, 2022 8:59:00 AM CEST        June 9, 2022 8:59:01 AM CEST        Success
Applying HMP Patches                     June 9, 2022 8:59:01 AM CEST        June 9, 2022 8:59:20 AM CEST        Success
Patch location validation                June 9, 2022 8:59:20 AM CEST        June 9, 2022 8:59:20 AM CEST        Success
oda-hw-mgmt upgrade                      June 9, 2022 8:59:20 AM CEST        June 9, 2022 8:59:52 AM CEST        Success
OSS Patching                             June 9, 2022 8:59:52 AM CEST        June 9, 2022 8:59:53 AM CEST        Success
Applying Firmware Disk Patches           June 9, 2022 8:59:53 AM CEST        June 9, 2022 8:59:57 AM CEST        Success
Applying Firmware Controller Patches     June 9, 2022 8:59:57 AM CEST        June 9, 2022 9:05:01 AM CEST        Success
Checking Ilom patch Version              June 9, 2022 9:05:01 AM CEST        June 9, 2022 9:05:01 AM CEST        Success
Patch location validation                June 9, 2022 9:05:01 AM CEST        June 9, 2022 9:05:01 AM CEST        Success
Save password in Wallet                  June 9, 2022 9:05:01 AM CEST        June 9, 2022 9:05:02 AM CEST        Success
Disabling IPMI v2                        June 9, 2022 9:05:02 AM CEST        June 9, 2022 9:05:02 AM CEST        Success
Apply Ilom patch                         June 9, 2022 9:05:02 AM CEST        June 9, 2022 9:05:02 AM CEST        Success
Copying Flash Bios to Temp location      June 9, 2022 9:05:02 AM CEST        June 9, 2022 9:05:02 AM CEST        Success
Starting the clusterware                 June 9, 2022 9:05:02 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
registering image                        June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
registering working copy                 June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
registering image                        June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
Creating GI home directories             June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
Extract GI clone                         June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
Provisioning Software Only GI with RHP   June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:06:49 AM CEST        Success
Patch GI with RHP                        June 9, 2022 9:06:49 AM CEST        June 9, 2022 9:15:56 AM CEST        Success
Updating GIHome version                  June 9, 2022 9:15:56 AM CEST        June 9, 2022 9:16:00 AM CEST        Success
Validate GI availability                 June 9, 2022 9:16:13 AM CEST        June 9, 2022 9:16:13 AM CEST        Success
Patch KVM CRS type                       June 9, 2022 9:16:13 AM CEST        June 9, 2022 9:16:15 AM CEST        Success
Patch VM vDisks CRS dependencies         June 9, 2022 9:16:15 AM CEST        June 9, 2022 9:16:15 AM CEST        Success
Update System version                    June 9, 2022 9:16:15 AM CEST        June 9, 2022 9:16:15 AM CEST        Success
Cleanup JRE Home                         June 9, 2022 9:16:15 AM CEST        June 9, 2022 9:16:15 AM CEST        Success
Add SYSNAME in Env                       June 9, 2022 9:16:15 AM CEST        June 9, 2022 9:16:15 AM CEST        Success
Starting the clusterware                 June 9, 2022 9:16:15 AM CEST        June 9, 2022 9:16:15 AM CEST        Success
Setting ACL for disk groups              June 9, 2022 9:16:15 AM CEST        June 9, 2022 9:16:19 AM CEST        Success
Update lvm.conf file                     June 9, 2022 9:18:09 AM CEST        June 9, 2022 9:18:09 AM CEST        Success
Update previous workarounds              June 9, 2022 9:18:09 AM CEST        June 9, 2022 9:18:09 AM CEST        Success
preRebootNode Actions                    June 9, 2022 9:18:09 AM CEST        June 9, 2022 9:21:14 AM CEST        Success
Reboot Ilom                              June 9, 2022 9:21:14 AM CEST        June 9, 2022 9:21:14 AM CEST        Success

Server reboots 5 minutes after the patch ends. On my X8-2M this server patching lasted 30 minutes.

Let’s check the component’s versions now:

odacli describe-component | grep -v ^$ 
System Version
---------------
19.15.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.15.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.15.0.0.0           up-to-date
GI
                                          19.15.0.0.220419      up-to-date
DB {
[OraDB19000_home1]
                                          19.12.0.0.210720      19.15.0.0.220419
[OraDB19000_home4 [wra]]
                                          19.14.0.0.220118      19.15.0.0.220419
[OraDB19000_home6 [roman]]
                                          19.14.0.0.220118      19.15.0.0.220419
[OraDB19000_home7 [DHE,MAW,TRSNOC,
RTSCDB1,RTSCDB2]]                         19.14.0.0.220118      19.15.0.0.220419
}
DCSCONTROLLER
                                          19.15.0.0.0           up-to-date
DCSCLI
                                          19.15.0.0.0           up-to-date
DCSAGENT
                                          19.15.0.0.0           up-to-date
DCSADMIN
                                          19.15.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c3]                                    80000681              up-to-date
[c4]                                    8000A87E              up-to-date
} 
SHARED CONTROLLER FIRMWARE VDV1RL04 up-to-date 
LOCAL DISK FIRMWARE            1132                up-to-date 
SHARED DISK FIRMWARE         1132                up-to-date 
HMP                                 2.4.8.0.601                up-to-date

This looks fine.

Patching the storage

Patching the storage is only needed for older ODAs/patch levels. In case you need to apply a patch on the storage it’s easy, there is a pre-patch, and then the patch:

odacli create-prepatchreport -st -v 19.15.0.0.0
odacli update-storage -v 19.15.0.0.0

For HA ODAs using RAC, patching can be done in a rolling fashion:

odacli update-storage -v 19.15.0.0.0 --rolling

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, I will apply the patch on the latest DB home only:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
6f78a962-22b9-4dc4-b14f-6e5c8c81f248     OraDB19000_home1     19.12.0.0.210720                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
940087c7-feb2-4e51-88f7-77f3dcacd0a7     OraDB19000_home4     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
adcf2c0d-7082-4ee0-9431-be331107f368     OraDB19000_home6     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED
0f2eed26-e7ca-4021-9329-902a858ce3a1     OraDB19000_home7     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_7 CONFIGURED

A prepatching is also needed here:

odacli create-prepatchreport -d -i 0f2eed26-e7ca-4021-9329-902a858ce3a1 -v 19.15.0.0.0
odacli describe-prepatchreport -i 38b56fb7-d582-450b-97ac-c515e43fd268

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  38b56fb7-d582-450b-97ac-c515e43fd268
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home7
                 Status:  FAILED
                Created:  June 9, 2022 9:57:47 AM CEST
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          0f2eed26-e7ca-4021-9329-902a858ce3a1
Validate patching tag           Success   Validated patching tag: 19.15.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01/app/odaorahome
Validate dbHomesOnACFS          Success   User has configured diskgroup for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days

I don’t care about Orachk recommendations on my databases as this is a test system. I will apply the patch on this DB home with the force option:

odacli update-dbhome -i 0f2eed26-e7ca-4021-9329-902a858ce3a1 -v 19.15.0.0.0 -f
odacli describe-job -i eaf625d9-0712-4c7f-a27f-d586a10f98e1

Job details
----------------------------------------------------------------
                     ID:  eaf625d9-0712-4c7f-a27f-d586a10f98e1
            Description:  DB Home Patching: Home Id is 0f2eed26-e7ca-4021-9329-902a858ce3a1
                 Status:  Success
                Created:  June 9, 2022 10:28:42 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Adding USER SSH_EQUIVALENCE              June 9, 2022 10:28:54 AM CEST       June 9, 2022 10:28:55 AM CEST       Success
Adding USER SSH_EQUIVALENCE              June 9, 2022 10:28:55 AM CEST       June 9, 2022 10:28:55 AM CEST       Success
Adding USER SSH_EQUIVALENCE              June 9, 2022 10:28:55 AM CEST       June 9, 2022 10:28:56 AM CEST       Success
Creating wallet for DB Client            June 9, 2022 10:29:34 AM CEST       June 9, 2022 10:29:34 AM CEST       Success
Patch databases by RHP                   June 9, 2022 10:29:34 AM CEST       June 9, 2022 10:36:53 AM CEST       Success
updating database metadata               June 9, 2022 10:36:53 AM CEST       June 9, 2022 10:36:53 AM CEST       Success
Set log_archive_dest for Database        June 9, 2022 10:36:53 AM CEST       June 9, 2022 10:36:57 AM CEST       Success
Patch databases by RHP                   June 9, 2022 10:36:57 AM CEST       June 9, 2022 10:44:57 AM CEST       Success
updating database metadata               June 9, 2022 10:44:58 AM CEST       June 9, 2022 10:44:58 AM CEST       Success
Set log_archive_dest for Database        June 9, 2022 10:44:58 AM CEST       June 9, 2022 10:45:01 AM CEST       Success
Patch databases by RHP                   June 9, 2022 10:45:01 AM CEST       June 9, 2022 10:49:22 AM CEST       Success
updating database metadata               June 9, 2022 10:49:22 AM CEST       June 9, 2022 10:49:22 AM CEST       Success
Set log_archive_dest for Database        June 9, 2022 10:49:22 AM CEST       June 9, 2022 10:49:25 AM CEST       Success
Update System version                    June 9, 2022 10:49:25 AM CEST       June 9, 2022 10:49:25 AM CEST       Success
TDE parameter update                     June 9, 2022 10:50:17 AM CEST       June 9, 2022 10:50:17 AM CEST       Success

A new DB home has been created and my databases are now linked to this new one:

odacli list-dbhomes


ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
6f78a962-22b9-4dc4-b14f-6e5c8c81f248     OraDB19000_home1     19.12.0.0.210720                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
940087c7-feb2-4e51-88f7-77f3dcacd0a7     OraDB19000_home4     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
adcf2c0d-7082-4ee0-9431-be331107f368     OraDB19000_home6     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED
0f2eed26-e7ca-4021-9329-902a858ce3a1     OraDB19000_home7     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_7 CONFIGURED
534017ac-c521-4929-a8f8-32a64d67fb8e     OraDB19000_home8     19.15.0.0.220419                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_8 CONFIGURED

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
83c90c99-feb2-4377-8d54-77c288e0ec93     roman      SI       19.14.0.0.220118     false      OLTP     odb1     ACFS       CONFIGURED   adcf2c0d-7082-4ee0-9431-be331107f368
44fa12df-c429-4aa5-ab60-f7f0d7856e18     wra        SI       19.14.0.0.220118     false      OLTP     odb1     ASM        CONFIGURED   940087c7-feb2-4e51-88f7-77f3dcacd0a7
e6be45b9-fe42-4035-9f15-3ee0ef8998e0     DHE        SI       19.15.0.0.220419     false      OLTP     odb1     ASM        CONFIGURED   534017ac-c521-4929-a8f8-32a64d67fb8e
dd6442a8-8593-4a17-8a7b-7430c828ad96     MAW        SI       19.15.0.0.220419     false      OLTP     odb1     ACFS       CONFIGURED   534017ac-c521-4929-a8f8-32a64d67fb8e
48a9de9d-18cc-4d7a-a84f-73d1f3f37973     TSYCDB1    SI       19.15.0.0.220419     true       OLTP     odb1s    ASM        CONFIGURED   534017ac-c521-4929-a8f8-32a64d67fb8e
fb8ec0cb-d9df-426c-9034-908100beba0b     TSYCDB2    SI       19.15.0.0.220419     true       OLTP     odb1s    ASM        CONFIGURED   534017ac-c521-4929-a8f8-32a64d67fb8e

The old DB home can now be removed safely:

odacli delete-dbhome -i 0f2eed26-e7ca-4021-9329-902a858ce3a1

If your databases were created with 19.11 or earlier versions, a parameter needs to be changed:

su - oracle
. oraenv <<< DBITST
sqlplus / as sysdba
alter system set "_enable_numa_support"=true scope=spfile sid='*';
exit
srvctl stop database -d DBITST_SITE1
srvctl start database -d DBITST_SITE1

This only concerns multi-processor ODAs (not S ones) and it will force an instance to use local memory modules, those associated to the processor where the instance is running. This should improve overall performance.

Patching the other DB homes is done the same way.

Remember that patching standby databases may raise an error, as datapatch cannot be applied on a mounted or read only database.

I would recommand to check on each primary the patch level after patching each DB home:

su – oracle
. oraenv <<< TSYCDB1
sqlplus / as sysdba
set serverout on
exec dbms_qopatch.get_sqlpatch_status;
...

Patch Id : 33808367
    Action : APPLY
    Action Time : 09-JUN-2022 10:44:46
    Description : OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)
    Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33808367/24680225/33808367_apply
_TSYCDB1_CDBROOT_2022Jun09_10_39_57.log
    Status : SUCCESS

Patch Id : 33806152
    Action : APPLY
    Action Time : 09-JUN-2022 10:44:46
    Description : Database Release Update : 19.15.0.0.220419 (33806152)
    Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33806152/24713297/33806152_apply
_TSYCDB1_CDBROOT_2022Jun09_10_39_57.log
    Status : SUCCESS

PL/SQL procedure successfully completed.
exit

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.15.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.15.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.15.0.0.0           up-to-date
GI
                                          19.15.0.0.220419      up-to-date
DB {
[OraDB19000_home1]
                                          19.12.0.0.210720      19.15.0.0.220419
[OraDB19000_home4 [wra]]
                                          19.14.0.0.220118      19.15.0.0.220419
[OraDB19000_home6 [roman]]
                                          19.14.0.0.220118      19.15.0.0.220419
[OraDB19000_home8 [DHE,MAW,RTSCDB1,
RTSCDB2]]                                 19.15.0.0.220419      up-to-date
}
DCSCONTROLLER
                                          19.15.0.0.0           up-to-date
DCSCLI
                                          19.15.0.0.0           up-to-date
DCSAGENT
                                          19.15.0.0.0           up-to-date
DCSADMIN
                                          19.15.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c3]                                   80000681               up-to-date
[c4]                                   8000A87E               up-to-date
} 
SHARED CONTROLLER FIRMWARE VDV1RL04 up-to-date 
LOCAL DISK FIRMWARE 1132 up-to-date 
SHARED DISK FIRMWARE 1132 up-to-date 
HMP 2.4.8.0.601 up-to-date

Everything is fine. I kept other DB homes in their old version.

Cleanse the old patches

The old patches will never be used again, so don’t forget to remove previous patch files from the repository if your ODA has already been patched:

odacli cleanup-patchrepo -cl -comp db,gi -v 19.14.0.0.0
odacli describe-job -i 7eb35bf8-f4dc-4afa-bd85-4193a574996a
Job details
----------------------------------------------------------------
          ID:  7eb35bf8-f4dc-4afa-bd85-4193a574996a
          Description:  Cleanup patchrepos
          Status:  Success
          Created:  June 9, 2022 11:03:14 AM CEST
          Message:
Task Name Start Time End Time Status
Cleanup Repository June 9, 2022 11:03:14 AM CEST June 9, 2022 11:03:14 AM CEST Success
Cleanup JRE Home June 9, 2022 11:03:14 AM CEST June 9, 2022 11:03:14 AM CEST Success
Cleanup old ASR rpm June 9, 2022 11:03:14 AM CEST June 9, 2022 11:03:14 AM CEST Success

Put back your own settings

  • add your additional rpms manually if needed
  • put back your profile scripts for grid and oracle users

Patching an existing DB System

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System.

odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
WSDBSYSTEMHD          odb2        10     16.00 GB    19.14.0.0.220118    19.14.0.0.220118    CONFIGURED       2022-05-05 11:57:47      2022-05-05 12:27:10
                                                                                                              CEST                     CEST
WSDBSYSTEM38          odb2        10     16.00 GB    19.14.0.0.220118    19.14.0.0.220118    CONFIGURED       2022-05-05 11:18:28      2022-05-05 11:47:36
                                                                                                              CEST                     CEST

In this example, I will need to connect to these 2 DB Systems and do the same upgrade I did on Bare Metal.

Remember that using multiple DB Systems is nice but it’s much more work when you need to patch.

Conclusion

This release is easy to apply coming from 19.14. So keep your ODA up-to-date, or eventually use the new DPR feature if you come from one of these old supported releases.