Introduction

Patch 19.17 is now available on Oracle Database Appliance. It’s time to test it.

What’s new?

This version brings latest PSUs to database and grid homes with their bug fixes, as usual. It also brings latest 21.8 databases but as DB Systems only, 21c being an innovation release. There are not that many new features and it’s not bad. No major OS upgrade, no breaking new feature, no big changes. At some points, most of us prefer stability and easy-to-apply patches.

The only new features I’ve read from the release note are:

  • Error correlation in the Browser User Interface (BUI), making a kind of crosscheck between odacli errors and log files from the various components. As you may know, odacli is just an interface on top of classic OS and Oracle tools, and it’s sometimes tough to find the log which refer to the error
  • Database patching can now be done at the database level, and not only at the DB home level, with oracle update-database
  • Enhanced patching when using Data Guard to limit downtime: I’m not so sure it will change something as it’s convenient to switch all the databases prior starting to patch

Which ODA is compatible with this 19.17?

The new ODAs X9-2S/L/HA and for sure X8, X7 and X6 series. X5-2HA is still on the compatibility list, so you can keep this old appliance up-to-date.

Is this patch a cumulative one?

This 19.17 can be applied on top of 19.13 or later. I just applied this new patch on an X7-2M running on 19.9 using 19.13 as an intermediate version.

Is there also a patch for my databases?

Only databases version 19c are now supported, and this is OK because this is the only one you should use now. Good bye 12.1. You may use Data Preserving Reprovisioning if you come from a very old version (<=18.8) and want to keep your 11g or 18c for example. I wrote a blog post on this feature a few months ago.

It’s not in the new feature list but it seems that it’s now possible to register an old DB clone without any problem. But I would recommend to only use 19c with this patch version. Using old databases is not supported, may not work properly with odacli, and you should be aware that 11g and 12cR1 are over now.

Download the patch and clone files

Download the patch and the corresponding clones to be able to apply the complete patch.

34753059 => the patch itself
30403673 => the GI clone needed for deploying newer GI version (mandatory)
30403662 => the DB clone for deploying new version of 19c

You don’t need the ISO file for patching, but I would recommend to download it (patch 30403643).

Be sure to choose the very latest 19.17 when downloading the clones, download link will first propose older versions.

Prepare the patching

Before running prepatch, please check these prerequisites:

  • filesystems have 20% available free space (does not concern acfs volumes)
  • additional rpms manually installed should be removed
  • revert profile scripts to default’s one (for grid and oracle users)
  • make sure you planned a generous downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much.

You should use odabr to make snapshots of the important filesystems prior patching. It’s fast and doesn’t cost anything. I would also recommend doing a backup of most important files: tnsnames.ora and listener.ora, database list, network configuration files, aso.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$

System Version
---------------
19.13.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB {
[ OraDB19000_home3,OraDB19000_home4 ]     19.13.0.0.211019      up-to-date
}
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      41100500              up-to-date
SHARED CONTROLLER FIRMWARE                QDV1RF32              up-to-date
LOCAL DISK FIRMWARE                       0121                  up-to-date
SHARED DISK FIRMWARE                      0121                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be updated with versions provided within the patch.

Prepararing the patch and updating the DCS tools

Copy the patch files on your ODA in a temp directory. On ODA X9-2 system disks are now smaller, so don’t hesitate to put the file on an nfs share or in the FRA acfs volume if you have acfs configured for your databases. Then unzip the files:

cd /u03/app/oracle
unzip -o p30403662_1917000_Linux-x86-64.zip  
unzip -o p30403673_1917000_Linux-x86-64.zip  
unzip -o p34753059_1917000_Linux-x86-64.zip

I first tried to register both patch and GI, but coming from 19.13 it doesn’t work:

odacli update-repository -f /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip,/u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip

DCS-10001:Internal error encountered: Cannot find the corresponding image for /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip in img_metadata.

It will work if you come from a newer version, from 19.16 for example.

But it’s not a problem as you can first register the patch only, update the DCS components and then register the GI clone:

odacli update-repository -f /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip
odacli describe-job -i "dbacdf31-405f-4f08-acf2-21d78b66945a"

Job details
----------------------------------------------------------------
                     ID:  dbacdf31-405f-4f08-acf2-21d78b66945a
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 9:55:39 AM CET
                Message:  /u03/app/oracle/oda-sm-19.17.0.0.0-221126.1-server.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 9:55:40 AM CET     December 2, 2022 9:56:09 AM CET     Success

odacli describe-component | grep -v ^$ 
System Version
---------------
19.13.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.17.0.0.0
GI                                        19.13.0.0.211019      19.17.0.0.221018
DB {
[ OraDB19000_home3,OraDB19000_home4 ]     19.13.0.0.211019      19.17.0.0.221018
}
DCSCONTROLLER                             19.13.0.0.0           19.17.0.0.0
DCSCLI                                    19.13.0.0.0           19.17.0.0.0
DCSAGENT                                  19.13.0.0.0           19.17.0.0.0
DCSADMIN                                  19.13.0.0.0           19.17.0.0.0
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      5.1.0.23.r146986
BIOS                                      41100500              41120100
SHARED CONTROLLER FIRMWARE                QDV1RF32              QDV1RF35
LOCAL DISK FIRMWARE                       0121                  not-available
SHARED DISK FIRMWARE                      0121                  not-available
HMP                                       2.4.8.0.600           2.4.8.9.603

Patching from 19.13 will normally be easy as there are not that many changes.

odacli update-dcsadmin -v 19.17.0.0.0
sleep 60; odacli describe-job -i "1b7ddd74-6108-4480-8584-251932476f44"
Job details
----------------------------------------------------------------
                     ID:  1b7ddd74-6108-4480-8584-251932476f44
            Description:  DcsAdmin patching
                 Status:  Success
                Created:  December 2, 2022 9:57:42 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                December 2, 2022 9:57:42 AM CET     December 2, 2022 9:57:42 AM CET     Success
dcs-admin upgrade                        December 2, 2022 9:57:42 AM CET     December 2, 2022 9:57:49 AM CET     Success

 
odacli update-dcscomponents -v 19.17.0.0.0
This job is interactive
{
  "jobId" : "c3ed2039-82b4-45a4-b7d4-813da9b2aa1a",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful.  dcscli RPM upgrade is successful.  dcscontroller RPM upgrade is successful.  Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful.  ",
  "reports" : null,
  "createTimestamp" : "December 02, 2022 10:03:37 AM CET",
  "description" : "Update-dcscomponents job completed and is not part of Agent job list",
  "updatedTime" : "December 02, 2022 10:05:16 AM CET"
}

odacli update-dcsagent -v 19.17.0.0.0
sleep 180; odacli describe-job -i "716a8259-1404-454e-90fb-cf59c14921d7"
Job details
----------------------------------------------------------------
                     ID:  716a8259-1404-454e-90fb-cf59c14921d7
            Description:  DcsAgent patching
                 Status:  Success
                Created:  December 2, 2022 10:06:21 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DCS Admin                           December 2, 2022 10:06:22 AM CET    December 2, 2022 10:06:22 AM CET    Success
Generate mTLS certificates               December 2, 2022 10:06:22 AM CET    December 2, 2022 10:06:23 AM CET    Success
Exporting Public Keys                    December 2, 2022 10:06:23 AM CET    December 2, 2022 10:06:24 AM CET    Success
Creating Trust Store                     December 2, 2022 10:06:24 AM CET    December 2, 2022 10:06:26 AM CET    Success
Update config files                      December 2, 2022 10:06:26 AM CET    December 2, 2022 10:06:26 AM CET    Success
Restart DCS Admin                        December 2, 2022 10:06:26 AM CET    December 2, 2022 10:06:46 AM CET    Success
Dcs-agent upgrade  to version 19.17.0.0.0 December 2, 2022 10:06:46 AM CET    December 2, 2022 10:07:58 AM CET    Success
Update System version                    December 2, 2022 10:07:58 AM CET    December 2, 2022 10:07:58 AM CET    Success

odacli update-repository -f /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip
sleep 60; odacli describe-job -i "4f81cf82-fed7-47a0-a6c3-69d3f9193e65"

Job details
----------------------------------------------------------------
                     ID:  4f81cf82-fed7-47a0-a6c3-69d3f9193e65
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 10:09:44 AM CET
                Message:  /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-GI-19.17.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 10:09:45 AM CET    December 2, 2022 10:10:21 AM CET    Success

Prepatching report

Let’s do the prepatching test:

odacli create-prepatchreport -s -v 19.17.0.0.0
sleep 600 ; odacli describe-prepatchreport -i bb50b38f-55be-424a-bdf9-7d70206a61e1

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  bb50b38f-55be-424a-bdf9-7d70206a61e1
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  FAILED
                Created:  December 2, 2022 10:12:55 AM CET
                 Result:  One or more pre-checks failed for [GI]

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
 
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Failed    Lock on central inventory detected:
                                          /u01/app/oraInventory/locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

On my configuration it didn’t work but this failure is a known issue:

rm -rf /u01/app/oraInventory/locks

odacli create-prepatchreport -s -v 19.17.0.0.0
sleep 600 ; odacli describe-prepatchreport -i d334c952-7bd8-4dc1-bf44-ee9247f7acc2

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d334c952-7bd8-4dc1-bf44-ee9247f7acc2
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
                 Status:  SUCCESS
                Created:  December 2, 2022 10:23:01 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
 
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Validate space for clones       Success   Clones volume is already created
volume
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

__SERVER__
Validate local patching         Success   Successfully validated server local
                                          patching
Validate command execution      Success   Validated command execution

Everything is OK to start patching.

Patching server and GI

Let’s start the update-server:

odacli update-server -v 19.17.0.0.0
odacli describe-job -i "6e32cf55-54b5-41fc-af07-0cb773e68132"
Job details
----------------------------------------------------------------
                     ID:  6e32cf55-54b5-41fc-af07-0cb773e68132
            Description:  Server Patching
                 Status:  Success
                Created:  December 2, 2022 10:35:33 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Validate ILOM server reachable           December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Validate DCS Admin mTLS setup            December 2, 2022 10:35:46 AM CET    December 2, 2022 10:35:46 AM CET    Success
Configure export clones resource         December 2, 2022 10:35:47 AM CET    December 2, 2022 10:35:48 AM CET    Success
Creating repositories using yum          December 2, 2022 10:35:48 AM CET    December 2, 2022 10:35:52 AM CET    Success
Updating YumPluginVersionLock rpm        December 2, 2022 10:35:52 AM CET    December 2, 2022 10:35:52 AM CET    Success
Applying OS Patches                      December 2, 2022 10:35:52 AM CET    December 2, 2022 10:46:25 AM CET    Success
Creating repositories using yum          December 2, 2022 10:46:26 AM CET    December 2, 2022 10:46:26 AM CET    Success
Applying HMP Patches                     December 2, 2022 10:46:26 AM CET    December 2, 2022 10:46:46 AM CET    Success
Patch location validation                December 2, 2022 10:46:46 AM CET    December 2, 2022 10:46:46 AM CET    Success
Oda-hw-mgmt upgrade                      December 2, 2022 10:46:47 AM CET    December 2, 2022 10:47:21 AM CET    Success
OSS Patching                             December 2, 2022 10:47:21 AM CET    December 2, 2022 10:47:21 AM CET    Success
Applying Firmware Disk Patches           December 2, 2022 10:47:21 AM CET    December 2, 2022 10:47:23 AM CET    Success
Applying Firmware Controller Patches     December 2, 2022 10:47:24 AM CET    December 2, 2022 10:48:12 AM CET    Success
Checking Ilom patch Version              December 2, 2022 10:48:12 AM CET    December 2, 2022 10:48:13 AM CET    Success
Patch location validation                December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:13 AM CET    Success
Save password in Wallet                  December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:13 AM CET    Success
Disabling IPMI v2                        December 2, 2022 10:48:13 AM CET    December 2, 2022 10:48:14 AM CET    Success
Apply Ilom patch                         December 2, 2022 10:48:14 AM CET    December 2, 2022 10:57:11 AM CET    Success
Copying Flash Bios to Temp location      December 2, 2022 10:57:11 AM CET    December 2, 2022 10:57:11 AM CET    Success
Starting the clusterware                 December 2, 2022 10:57:11 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering image                        December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering working copy                 December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Registering image                        December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Creating GI home directories             December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:54 AM CET    Success
Extract GI clone                         December 2, 2022 10:58:54 AM CET    December 2, 2022 10:58:55 AM CET    Success
Provisioning Software Only GI with RHP   December 2, 2022 10:58:55 AM CET    December 2, 2022 10:58:55 AM CET    Success
Patch GI with RHP                        December 2, 2022 10:58:55 AM CET    December 2, 2022 11:05:10 AM CET    Success
Updating GIHome version                  December 2, 2022 11:05:10 AM CET    December 2, 2022 11:05:14 AM CET    Success
Validate GI availability                 December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch KVM CRS type                       December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch VM vDisks CRS dependencies         December 2, 2022 11:05:52 AM CET    December 2, 2022 11:05:52 AM CET    Success
Patch DB System domain config            December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Update System version                    December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Cleanup JRE Home                         December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Add SYSNAME in Env                       December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Starting the clusterware                 December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:53 AM CET    Success
Setting ACL for disk groups              December 2, 2022 11:05:53 AM CET    December 2, 2022 11:05:57 AM CET    Success
Enable LKCE                              December 2, 2022 11:07:50 AM CET    December 2, 2022 11:10:37 AM CET    Success
Update previous workarounds              December 2, 2022 11:10:37 AM CET    December 2, 2022 11:10:37 AM CET    Success
Generating and saving BOM                December 2, 2022 11:10:37 AM CET    December 2, 2022 11:12:32 AM CET    Success
PreRebootNode Actions                    December 2, 2022 11:12:32 AM CET    December 2, 2022 11:13:19 AM CET    Success
Reboot Ilom                              December 2, 2022 11:13:19 AM CET    December 2, 2022 11:13:19 AM CET    Success

Server reboots 5 minutes after the patch ends. On this X7-2M server patching lasted 40 minutes.

Checking filesystems told me that there is a problem on the /boot. It’s quite common on an ODA where multiple patches were applied. If you have this problem, please identify the old kernels and remove the rpms packages:

df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        474M  451M     0 100% /boot

uname -a
Linux uns-oda2 4.14.35-2047.518.4.1.el7uek.x86_64 #2 SMP Tue Oct 18 18:08:21 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
rpm -qa | grep kernel-uek-4
kernel-uek-4.14.35-2047.508.3.2.el7uek.x86_64
kernel-uek-4.14.35-2025.400.9.el7uek.x86_64
kernel-uek-4.14.35-2047.518.4.1.el7uek.x86_64

rpm -e kernel-uek-4.14.35-2025.400.9.el7uek.x86_64

df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        474M  331M  119M  74% /boot

Patching the storage

Patching the storage is only needed if describe-component tells you that you’re not up-to-date. On my X7-2M I had to patch:

odacli update-storage -v 19.17.0.0.0
odacli describe-job -i "93199b6d-b400-43cf-812f-20ffaf59d704"
Job details
----------------------------------------------------------------
                     ID:  93199b6d-b400-43cf-812f-20ffaf59d704
            Description:  Storage Firmware Patching
                 Status:  Success
                Created:  December 2, 2022 11:29:47 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches           December 2, 2022 11:29:53 AM CET    December 2, 2022 11:29:55 AM CET    Success
Applying Firmware Controller Patches     December 2, 2022 11:29:56 AM CET    December 2, 2022 11:37:21 AM CET    Success
Generating and saving BOM                December 2, 2022 11:37:21 AM CET    December 2, 2022 11:38:28 AM CET    Success
PreRebootNode Actions                    December 2, 2022 11:38:28 AM CET    December 2, 2022 11:38:28 AM CET    Success
Reboot Ilom                              December 2, 2022 11:38:28 AM CET    December 2, 2022 11:38:28 AM CET    Success

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, I will apply the patch on my two 19c DB homes:

odacli list-dbhomes
ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
f33e91e6-ac8c-48b1-a599-2e32ddf46b30     OraDB19000_home3     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
804165be-d58e-44c6-b51d-c2729a99b8e9     OraDB19000_home4     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED

The DB clone is needed:

cd /u03/app/oracle/
unzip -o p30403662_1917000_Linux-x86-64.zip
odacli update-repository -f /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip
odacli describe-job -i "5ab4fcaa-2452-4f43-b329-cc3df03abbf7"

Job details
----------------------------------------------------------------
                     ID:  5ab4fcaa-2452-4f43-b329-cc3df03abbf7
            Description:  Repository Update
                 Status:  Success
                Created:  December 2, 2022 11:50:02 AM CET
                Message:  /u03/app/oracle/odacli-dcs-19.17.0.0.0-221029-DB-19.17.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 2, 2022 11:50:02 AM CET    December 2, 2022 11:50:38 AM CET    Success

A prepatching is also needed here:

odacli create-prepatchreport -d -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30 -v 19.17.0.0.0
odacli describe-prepatchreport -i 4142cc7e-edbf-4aa2-9db1-cf5f2ea2ab94

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  4142cc7e-edbf-4aa2-9db1-cf5f2ea2ab94
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home3
                 Status:  FAILED
                Created:  December 2, 2022 11:55:08 AM CET
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
uns-oda2

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          f33e91e6-ac8c-48b1-a599-2e32ddf46b30
Validate patching tag           Success   Validated patching tag: 19.17.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01
Validate dbHomesOnACFS          Success   User has configured disk group for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP.  and local patching
                                          is possible
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    ORAchk validation failed: .
Validate command execution      Success   Validated command execution
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value
Check for parameter             Failed    AHF-3744: Database parameter
global_names                              GLOBAL_NAMES is not set to
                                          recommended value

I don’t care about Orachk recommendations on my databases because it has been set like that. I will apply the patch on this DB home with the force option:

odacli update-dbhome -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30 -v 19.17.0.0.0 -f
sleep 600 ; odacli describe-job -i "0e4a424b-9892-46d9-90d8-ebb61737ef03"

Job details
----------------------------------------------------------------
                     ID:  0e4a424b-9892-46d9-90d8-ebb61737ef03
            Description:  DB Home Patching: Home Id is f33e91e6-ac8c-48b1-a599-2e32ddf46b30
                 Status:  Success
                Created:  December 2, 2022 12:11:31 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating wallet for DB Client            December 2, 2022 12:12:18 PM CET    December 2, 2022 12:12:18 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:12:18 PM CET    December 2, 2022 12:18:34 PM CET    Success
Updating database metadata               December 2, 2022 12:18:34 PM CET    December 2, 2022 12:18:34 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:18:34 PM CET    December 2, 2022 12:18:37 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:18:37 PM CET    December 2, 2022 12:20:09 PM CET    Success
Updating database metadata               December 2, 2022 12:20:09 PM CET    December 2, 2022 12:20:09 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:20:09 PM CET    December 2, 2022 12:20:13 PM CET    Success
Patch databases by RHP                   December 2, 2022 12:20:13 PM CET    December 2, 2022 12:25:58 PM CET    Success
Updating database metadata               December 2, 2022 12:25:58 PM CET    December 2, 2022 12:25:58 PM CET    Success
Set log_archive_dest for Database        December 2, 2022 12:25:58 PM CET    December 2, 2022 12:26:00 PM CET    Success
Update System version                    December 2, 2022 12:26:00 PM CET    December 2, 2022 12:26:00 PM CET    Success
Generating and saving BOM                December 2, 2022 12:26:00 PM CET    December 2, 2022 12:28:03 PM CET    Success
TDE parameter update                     December 2, 2022 12:28:39 PM CET    December 2, 2022 12:28:39 PM CET    Success

It’s the exact same procedure for the second DB home.

New DB homes have been created and my databases are now linked to these new ones:

odacli list-dbhomes
ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
f33e91e6-ac8c-48b1-a599-2e32ddf46b30     OraDB19000_home3     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGURED
804165be-d58e-44c6-b51d-c2729a99b8e9     OraDB19000_home4     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_4 CONFIGURED
7741bb27-43cc-484c-9525-5f854262c33d     OraDB19000_home5     19.17.0.0.221018                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_5 CONFIGURED
cacbc994-04ae-4f40-ae96-a5b57bb3d89a     OraDB19000_home6     19.17.0.0.221018                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_6 CONFIGURED

odacli list-databases
ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
cbc31d02-5507-4e90-97b4-42a65bb2e2b7     PCGO_LCDF  SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d
118c79a4-c5cc-463c-ae29-d92208f502a7     PCSGO_LCDF SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   cacbc994-04ae-4f40-ae96-a5b57bb3d89a
4892fc73-e89a-4fc4-8fac-c4d4f91ba64d     TCGO       SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d
ef0cc1fe-86b8-4f6d-9abe-a619147337ca     TCSGO      SI       19.17.0.0.221018     false      OLTP     odb1     ACFS       CONFIGURED   7741bb27-43cc-484c-9525-5f854262c33d

The old DB homes can now be safely removed:

odacli delete-dbhome -i f33e91e6-ac8c-48b1-a599-2e32ddf46b30
sleep 60 ;  odacli delete-dbhome -i 804165be-d58e-44c6-b51d-c2729a99b8e9
sleep 120 ; odacli list-jobs | tail -n 3
212c0662-a062-40c6-925b-3b915effa23d     Database Home OraDB19000_home3 Deletion with id f33e91e6-ac8c-48b1-a599-2e32ddf46b30 December 2, 2022 1:01:09 PM CET     Success
a64b32ac-9cad-4f23-acec-66077a5f856d     Database Home OraDB19000_home4 Deletion with id 804165be-d58e-44c6-b51d-c2729a99b8e9 December 2, 2022 1:01:41 PM CET     Success

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.17.0.0.0
System node Name
---------------
uns-oda2
Local System Version
---------------
19.17.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.17.0.0.0           up-to-date
GI
                                          19.17.0.0.221018      up-to-date
DB {
[OraDB19000_home5 [PCGO_LCDF,TCGO,
TCSGO]]                                   19.17.0.0.221018      up-to-date
[OraDB19000_home6 [PCSGO_LCDF]]
                                          19.17.0.0.221018      up-to-date
}
DCSCONTROLLER
                                          19.17.0.0.0           up-to-date
 
DCSCLI
                                          19.17.0.0.0           up-to-date
DCSAGENT
                                          19.17.0.0.0           up-to-date
DCSADMIN
                                          19.17.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.1.0.23.r146986      up-to-date
BIOS
                                          41120100              up-to-date
LOCAL CONTROLLER FIRMWARE {
[c6]
                                          80000690              up-to-date
[c7]
                                          214.2.271.9           up-to-date
}
SHARED CONTROLLER FIRMWARE
                                          QDV1RF35              up-to-date
LOCAL DISK FIRMWARE
                                          N2010121              up-to-date
SHARED DISK FIRMWARE
                                          N2010121              up-to-date
HMP
                                          2.4.8.9.603           up-to-date

Everything is fine. I’m quiet for 2-3 months.

Cleanse the old patches

The old patches will never be used again, so don’t forget to remove them from the repository if your ODA has already been patched:

odacli cleanup-patchrepo -cl -v 19.13.0.0.0

odacli describe-job -i "7e1506b9-5778-47ed-84eb-65d976819c80"
Job details
----------------------------------------------------------------
                     ID:  7e1506b9-5778-47ed-84eb-65d976819c80
            Description:  Cleanup patchrepos
                 Status:  Success
                Created:  December 2, 2022 12:56:14 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Cleanup Repository                       December 2, 2022 12:56:14 PM CET    December 2, 2022 12:56:14 PM CET    Success
Cleanup JRE Home                         December 2, 2022 12:56:15 PM CET    December 2, 2022 12:56:15 PM CET    Success
Cleanup old ASR rpm                      December 2, 2022 12:56:15 PM CET    December 2, 2022 12:56:15 PM CET    Success

Put back your own settings

Once everything is OK, don’t forget to put back your settings:

  • add your additional rpms manually if needed
  • put back your profile scripts for grid and oracle users

Patching DB Systems

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. I haven’t tried on my X7-2M because it only runs bare metal databases, but you will need to list your DB Systems, connect to each one, and apply the server and database patches:

odacli list-dbsystems
ssh ...
odacli update-server -v 19.17.0.0.0
odacli update-dbhome -i ... -v 19.17.0.0.0

Conclusion

This release is easy to apply coming from 19.13 and also probably from newer versions. Future releases may include a major OS upgrade and 23c sooner or later.