Introduction

Patch 19.14 is now available on Oracle Database Appliance. It’s time to test it.

What’s new?

This version brings January’s PSU to database and grid homes with their bug fixes, as usual. It also brings latest 21.5 but only for DB Systems (21c being an innovation release). Don’t expect too many new features for this release (actually, I don’t see anything new), it’s much more a bundle of bug fixes and finally it’s what I like.

Most important new features were brought by previous 19.13: multi-user mode for odacli tasks and parameter repository for bare metal instances.

Which ODA is compatible with this 19.14?

ODA X8 is becoming quite old now but no X9 has been announced yet. So the X8 in its 3 flavors (S, M, HA) is compatible with this patch, so are X7, X6 and X5-2HA (probably one of the latest patch for this one).

Is this patch a cumulative one?

This 19.14 can be applied on top of 19.10 or later. If you’re using older version like 19.6 or 19.8, applying 19.10 is needed prior 19.14.

As usual, don’t hesitate to consider reimaging instead of patching if 3 or more patches are needed to reach 19.14.

Is there also a patch for my databases?

If you’re using 12.1, 12.2 or 19c, these databases can be patched. 11gR2 and 18c are no more supported on ODA, meaning that your databases will continue working but you should definitely consider an upgrade. If you choose to reimage to this latest version, you will not find any 18c or 11gR2 DB homes to deploy.

Download the patch and clone files

Download the patch and the corresponding clones to be able to apply the complete patch.

  • 33702951 => the patch itself
  • 30403673 => the GI clone needed for deploying new version (mandatory)
  • 30403662 => the DB clone for deploying new version of 19c (if you use 19c databases)
  • 27119402 => the DB clone for deploying new version of 12.2 (if you use 12.2 databases)
  • 23494992 => the DB clone for deploying new version of 12.1 (if you use 12.1 databases)

Be sure to choose the very latest 19.14 when downloading the clones, download link will first propose older versions.

Prepare the patching

A pre-patch is now provided and needed before applying an ODA patch, but prior running it, please check these prerequisites:

  • filesystems have 20% available free space (does not concern acfs volumes)
  • additional rpms manually installed should be removed
  • revert profile scripts to default’s one (concerns grid and oracle users)
  • make sure you can afford longer than planned downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much

You can use odabr to backup your filesystems to snapshot or to nfs, or simply backup all your important files to a nfs share in case patching would fail.

Version precheck

Start to check current version on all components:

odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           up-to-date
GI                                        19.13.0.0.211019      up-to-date
DB                                        19.13.0.0.211019      up-to-date
DCSCONTROLLER                             19.13.0.0.0           up-to-date
DCSCLI                                    19.13.0.0.0           up-to-date
DCSAGENT                                  19.13.0.0.0           up-to-date
DCSADMIN                                  19.13.0.0.0           up-to-date
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Once the patch will be registered in the ODA repository, the “Available Version” column will be fed with versions provided within the patch.

Patching from 19.13 will normally be easy.

Prepararing the patch files and updating the DCS tools

Copy the patch files on your ODA in a temp directory. Then unzip the files:

cd /opt/dbi/
for f in p*1914000*.zip; do unzip -n $f; done
Archive:  p30403662_1914000_Linux-x86-64.zip
 extracting: odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip
Archive:  p30403673_1914000_Linux-x86-64.zip
 extracting: odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip
Archive:  p33702951_1914000_Linux-x86-64.zip
 extracting: oda-sm-19.14.0.0.0-220215-server.zip

rm -rf p*1914000*.zip

Register the patch in the repository:

odacli update-repository -f /opt/dbi/oda-sm-19.14.0.0.0-220215-server.zip

odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.14.0.0.0
GI                                        19.13.0.0.211019      19.14.0.0.220118
DB                                        19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER                             19.13.0.0.0           19.14.0.0.0
DCSCLI                                    19.13.0.0.0           19.14.0.0.0
DCSAGENT                                  19.13.0.0.0           19.14.0.0.0
DCSADMIN                                  19.13.0.0.0           19.14.0.0.0
OS                                        7.9                   up-to-date
ILOM                                      5.0.2.24.r141466      up-to-date
BIOS                                      52050300              up-to-date
SHARED CONTROLLER FIRMWARE                VDV1RL04              up-to-date
LOCAL DISK FIRMWARE                       1132                  up-to-date
SHARED DISK FIRMWARE                      1132                  up-to-date
HMP                                       2.4.8.0.600           up-to-date

Update the DCS tooling of your ODA:

odacli update-dcsadmin -v 19.14.0.0.0
sleep 30;  odacli update-dcscomponents -v 19.14.0.0.0
odacli update-dcsagent -v 19.14.0.0.0

Note that updating the DCS components is not done through a job:

sleep 180; odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 4
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
0526db5b-69cd-42c6-b8af-93d34c7c261f     Repository Update                                                           March 1, 2022 9:31:20 AM CET        Success
bdf6a313-2e0c-4359-adb6-0e7606c38735     DcsAdmin patching                                                           March 1, 2022 9:32:52 AM CET        Success
b9905530-d1ad-41ec-9b52-94f62728d4b5     DcsAgent patching                                                           March 1, 2022 9:34:49 AM CET        Success

Now you can register GI and DB clones:

odacli update-repository -f /opt/dbi/odacli-dcs-19.14.0.0.0-220215-GI-19.14.0.0.zip
sleep 50; odacli update-repository -f /opt/dbi/odacli-dcs-19.14.0.0.0-220215-DB-19.14.0.0.zip

sleep 50; odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 3
ID                                       Description                                                                 Created                             Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
d72451bf-c0da-42b8-8fd9-72d45af5aabc     Repository Update                                                           March 1, 2022 9:39:00 AM CET        Success
b94fcecc-549c-4011-b239-022a88aa68e2     Repository Update                                                           March 1, 2022 9:40:04 AM CET        Success

odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.13.0.0.0           19.14.0.0.0
GI
                                          19.13.0.0.211019      19.14.0.0.220118
DB
                                          19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

This update will be limited to Oracle software.

Pre-patching report

Let’s check if patching has the green light:

odacli create-prepatchreport -s -v 19.14.0.0.0

odacli describe-prepatchreport -i 71867a62-37aa-4b95-a841-893977038cca

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  71867a62-37aa-4b95-a841-893977038cca
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  March 1, 2022 9:52:45 AM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate ILOM server reachable  Success   Successfully connected with ILOM
                                          server using public IP and USB
                                          interconnect
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

Everything is OK to start patching.

Patching infrastructure and GI

Let’s start the update-server:

odacli update-server -v 19.14.0.0.0
odacli describe-job -i 8e77a48d-672e-4b81-a6dc-6e396c32e99c

Job details
----------------------------------------------------------------
                     ID:  8e77a48d-672e-4b81-a6dc-6e396c32e99c
            Description:  Server Patching
                 Status:  Success
                Created:  March 1, 2022 10:11:30 AM CET
                Message:  Successfully patched GI with RHP

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata              March 1, 2022 10:11:42 AM CET       March 1, 2022 10:11:42 AM CET       Success
Validate ILOM server reachable           March 1, 2022 10:11:42 AM CET       March 1, 2022 10:11:42 AM CET       Success
Configure export clones resource         March 1, 2022 10:11:43 AM CET       March 1, 2022 10:11:44 AM CET       Success
Creating repositories using yum          March 1, 2022 10:11:44 AM CET       March 1, 2022 10:11:46 AM CET       Success
Updating YumPluginVersionLock rpm        March 1, 2022 10:11:46 AM CET       March 1, 2022 10:11:47 AM CET       Success
Applying OS Patches                      March 1, 2022 10:11:47 AM CET       March 1, 2022 10:18:26 AM CET       Success
Creating repositories using yum          March 1, 2022 10:18:27 AM CET       March 1, 2022 10:18:27 AM CET       Success
Applying HMP Patches                     March 1, 2022 10:18:27 AM CET       March 1, 2022 10:18:28 AM CET       Success
Patch location validation                March 1, 2022 10:18:28 AM CET       March 1, 2022 10:18:28 AM CET       Success
oda-hw-mgmt upgrade                      March 1, 2022 10:18:28 AM CET       March 1, 2022 10:18:57 AM CET       Success
OSS Patching                             March 1, 2022 10:18:57 AM CET       March 1, 2022 10:18:57 AM CET       Success
Applying Firmware Disk Patches           March 1, 2022 10:18:57 AM CET       March 1, 2022 10:19:01 AM CET       Success
Applying Firmware Controller Patches     March 1, 2022 10:19:01 AM CET       March 1, 2022 10:19:05 AM CET       Success
Checking Ilom patch Version              March 1, 2022 10:19:05 AM CET       March 1, 2022 10:19:05 AM CET       Success
Patch location validation                March 1, 2022 10:19:05 AM CET       March 1, 2022 10:19:05 AM CET       Success
Save password in Wallet                  March 1, 2022 10:19:05 AM CET       March 1, 2022 10:19:05 AM CET       Success
Disabling IPMI v2                        March 1, 2022 10:19:06 AM CET       March 1, 2022 10:19:06 AM CET       Success
Apply Ilom patch                         March 1, 2022 10:19:06 AM CET       March 1, 2022 10:19:06 AM CET       Success
Copying Flash Bios to Temp location      March 1, 2022 10:19:06 AM CET       March 1, 2022 10:19:06 AM CET       Success
Starting the clusterware                 March 1, 2022 10:19:06 AM CET       March 1, 2022 10:20:48 AM CET       Success
registering image                        March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
registering working copy                 March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
registering image                        March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Creating GI home directories             March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Extract GI clone                         March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Provisioning Software Only GI with RHP   March 1, 2022 10:20:49 AM CET       March 1, 2022 10:20:49 AM CET       Success
Patch GI with RHP                        March 1, 2022 10:20:49 AM CET       March 1, 2022 10:30:06 AM CET       Success
Updating GIHome version                  March 1, 2022 10:30:06 AM CET       March 1, 2022 10:30:09 AM CET       Success
Validate GI availability                 March 1, 2022 10:30:12 AM CET       March 1, 2022 10:30:12 AM CET       Success
Patch KVM CRS type                       March 1, 2022 10:30:12 AM CET       March 1, 2022 10:30:13 AM CET       Success
Update System version                    March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Cleanup JRE Home                         March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Add SYSNAME in Env                       March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Starting the clusterware                 March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:13 AM CET       Success
Setting ACL for disk groups              March 1, 2022 10:30:13 AM CET       March 1, 2022 10:30:17 AM CET       Success
Update lvm.conf file                     March 1, 2022 10:31:34 AM CET       March 1, 2022 10:31:34 AM CET       Success
Update previous workarounds              March 1, 2022 10:31:34 AM CET       March 1, 2022 10:31:34 AM CET       Success
preRebootNode Actions                    March 1, 2022 10:31:34 AM CET       March 1, 2022 10:36:07 AM CET       Success
Reboot Ilom                              March 1, 2022 10:36:07 AM CET       March 1, 2022 10:36:07 AM CET       Success

Server reboots 5 minutes after the patch ends. On my X8-2M this server patching lasted 25 minutes.

Let’s check the component’s versions now:

odacli describe-component | grep -v ^$ 
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB
                                          19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

This looks fine.

Patching the storage

Patching the storage is only needed for older ODAs/patch levels. In case you need to apply a patch on the storage it’s easy: there is a pre-patch, and then the patch:

odacli create-prepatchreport -st -v 19.14.0.0.0
odacli update-storage -v 19.14.0.0.0

For HA ODAs using RAC, patching can be done in a rolling fashion:

odacli update-storage -v 19.14.0.0.0 --rolling

I never encountered troubles during storage patching, so it should be fine.

Patching the DB homes

Since 19.11, DB homes are created on an acfs filesystem. If you come from 19.10, you will need to configure this filesystem:

odacli configure-dbhome-storage -dg DATA

Time for patching the DB homes depends on the number of DB homes and number of databases. In this example, only one DB home is deployed:

odacli list-dbhomes

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
5e7a146a-c18d-427b-a206-e532ec9907cf     OraDB19000_home1     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED

A prepatching is also needed here:

odacli create-prepatchreport -d -i 5e7a146a-c18d-427b-a206-e532ec9907cf -v 19.14.0.0.0
odacli describe-prepatchreport -i d7aa02c8-cd13-4729-891d-ea52fade3784

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d7aa02c8-cd13-4729-891d-ea52fade3784
            Description:  Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
                 Status:  FAILED
                Created:  March 1, 2022 10:50:55 AM CET
                 Result:  One or more pre-checks failed for [ORACHK]

Node Name
---------------
dbi-oda-x8

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID             Success   Validated DB Home ID:
                                          5e7a146a-c18d-427b-a206-e532ec9907cf
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate minimum agent version  Success   Validated minimum agent version
Is GI upgraded                  Success   Validated GI is upgraded
Validate available space for    Success   Validated free space required under
db                                        /u01/app/odaorahome
Validate dbHomesOnACFS          Success   User has configured diskgroup for
configured                                Database homes on ACFS
Validate Oracle base            Success   Successfully validated Oracle Base
Is DB clone available           Success   Successfully validated clone file
                                          exists
Evaluate DBHome patching with   Success   Successfully validated updating
RHP                                       dbhome with RHP
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Failed    Orachk validation failed: .
Validate command execution      Success   Validated command execution
Verify the Fast Recovery Area   Failed    AHF-2929: FRA space management
(FRA) has reclaimable space               problem file types are present
                                          without an RMAN backup completion
                                          within the last 7 days

I don’t care about Orachk recommendations on my database as this is a test system. I will apply the patch on this DB home:

odacli update-dbhome -i 5e7a146a-c18d-427b-a206-e532ec9907cf -v 19.14.0.0.0 -f
DCS-10802:Insufficient disk space on file system: /u01/app/odaorahome. Expected free space: 16GB, available space: 5GB

Default acfs filesystem size for db homes is quite small, and it does not extent automatically, so let’s increase its size:

odacli list-dbhome-storages


=============================================================================================================
ID                                     Node Description          Disk Group Volume      Size(GB)   Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
9f091fa4-a8de-4c02-9504-4c889c324598   0    ORACLE_HOME          DATA       orahome_sh  30         CONFIGURED
09747e05-d7d4-4f5d-94ab-b1eb7c730c74   0    ORACLE_BASE          DATA       odabase_n0  10         CONFIGURED
=============================================================================================================

odacli modify-dbhome-storage -i 9f091fa4-a8de-4c02-9504-4c889c324598 -s 60

odacli list-dbhome-storages


=============================================================================================================
ID                                     Node Description          Disk Group Volume      Size(GB)   Status
-------------------------------------- ---- -------------------- ---------- ----------- ---------- ----------
9f091fa4-a8de-4c02-9504-4c889c324598   0    ORACLE_HOME          DATA       orahome_sh  60         CONFIGURED
09747e05-d7d4-4f5d-94ab-b1eb7c730c74   0    ORACLE_BASE          DATA       odabase_n0  10         CONFIGURED
=============================================================================================================


odacli update-dbhome -i 5e7a146a-c18d-427b-a206-e532ec9907cf -v 19.14.0.0.0 -f

odacli describe-job -i 9293caea-6521-4e62-991d-cf66be1e10cf

Job details
----------------------------------------------------------------
                     ID:  9293caea-6521-4e62-991d-cf66be1e10cf
            Description:  DB Home Patching: Home Id is 5e7a146a-c18d-427b-a206-e532ec9907cf
                 Status:  Success
                Created:  March 1, 2022 11:09:06 AM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Adding USER SSH_EQUIVALENCE              March 1, 2022 11:09:12 AM CET       March 1, 2022 11:09:12 AM CET       Success
Adding USER SSH_EQUIVALENCE              March 1, 2022 11:09:12 AM CET       March 1, 2022 11:09:12 AM CET       Success
Adding USER SSH_EQUIVALENCE              March 1, 2022 11:09:12 AM CET       March 1, 2022 11:09:13 AM CET       Success
Creating wallet for DB Client            March 1, 2022 11:09:49 AM CET       March 1, 2022 11:09:49 AM CET       Success
Patch databases by RHP                   March 1, 2022 11:09:49 AM CET       March 1, 2022 11:13:06 AM CET       Success
updating database metadata               March 1, 2022 11:13:06 AM CET       March 1, 2022 11:13:06 AM CET       Success
Set log_archive_dest for Database        March 1, 2022 11:13:06 AM CET       March 1, 2022 11:13:09 AM CET       Success
Patch databases by RHP                   March 1, 2022 11:13:09 AM CET       March 1, 2022 11:17:19 AM CET       Success
updating database metadata               March 1, 2022 11:17:19 AM CET       March 1, 2022 11:17:19 AM CET       Success
Set log_archive_dest for Database        March 1, 2022 11:17:19 AM CET       March 1, 2022 11:17:22 AM CET       Success
Patch databases by RHP                   March 1, 2022 11:17:22 AM CET       March 1, 2022 11:22:08 AM CET       Success
updating database metadata               March 1, 2022 11:22:08 AM CET       March 1, 2022 11:22:08 AM CET       Success
Set log_archive_dest for Database        March 1, 2022 11:22:08 AM CET       March 1, 2022 11:22:10 AM CET       Success
Update System version                    March 1, 2022 11:22:10 AM CET       March 1, 2022 11:22:10 AM CET       Success
TDE parameter update                     March 1, 2022 11:23:02 AM CET       March 1, 2022 11:23:02 AM CET       Success

A new DB home has been created and my databases are now linked to this new one:

odacli list-dbhomes


ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
5e7a146a-c18d-427b-a206-e532ec9907cf     OraDB19000_home1     19.13.0.0.211019                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
530ce9aa-195c-4b45-98d4-5ab03855d47d     OraDB19000_home2     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED

odacli list-databases

ID                                       DB Name    DB Type  DB Version           CDB        Class    Shape    Storage    Status        DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
42656d90-e18e-4325-a69a-13d3a6b58710     DBITST     SI       19.14.0.0.220118     true       OLTP     odb1     ASM        CONFIGURED   530ce9aa-195c-4b45-98d4-5ab03855d47d
a2193c80-d004-4295-847e-ec8277952bd7     DBTAG38    SI       19.14.0.0.220118     false      OLTP     odb1     ASM        CONFIGURED   530ce9aa-195c-4b45-98d4-5ab03855d47d
e7f9bf31-d185-4159-b579-181c75d6835b     CDBCBL     SI       19.14.0.0.220118     true       OLTP     odb1     ACFS       CONFIGURED   530ce9aa-195c-4b45-98d4-5ab03855d47d

The old DB home can now be removed safely:

odacli delete-dbhome -i 5e7a146a-c18d-427b-a206-e532ec9907cf

If your databases were created with 19.11 or earlier versions, a parameter needs to be changed:

su - oracle
. oraenv <<< DBITST
sqlplus / as sysdba
alter system set "_enable_numa_support"=true scope=spfile sid='*';
exit
srvctl stop database -d DBITST_SITE1
srvctl start database -d DBITST_SITE1

This only concerns multi-processor ODAs (not S ones) and it will force an instance to use local memory modules, those associated to the processor where the instance is running. This should improve overall performance.

Patching the other DB homes is done the same way.

Remember that patching standby databases may raise an error, as datapatch cannot be applied on a mounted or read only database.

I would recommand to check on each primary the patch level after patching each db home:

su – oracle
. oraenv <<< DBITST
sqlplus / as sysdba
set serverout on
exec dbms_qopatch.get_sqlpatch_status;
Patch Id : 33192694
	Action : APPLY
	Action Time : 10-NOV-2021 06:45:49
	Description : OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)
	Logfile :
/u01/app/oracle/cfgtoollogs/sqlpatch/33192694/24421575/33192694_apply_G19794_CDB
ROOT_2021Nov10_06_29_39.log
	Status : SUCCESS

Patch Id : 33192793
	Action : APPLY
	Action Time : 10-NOV-2021 06:45:49
	Description : Database Release Update : 19.13.0.0.211019 (33192793)
	Logfile :
/u01/app/oracle/cfgtoollogs/sqlpatch/33192793/24462514/33192793_apply_G19794_CDB
ROOT_2021Nov10_06_29_39.log
	Status : SUCCESS

Patch Id : 33192694
	Action : ROLLBACK
	Action Time : 01-MAR-2022 11:17:15
	Description : OJVM RELEASE UPDATE: 19.13.0.0.211019 (33192694)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33192694/24421575/33192694_rollb
ack_DBITST_CDBROOT_2022Mar01_11_15_15.log
	Status : SUCCESS

Patch Id : 33561310
	Action : APPLY
	Action Time : 01-MAR-2022 11:17:16
	Description : OJVM RELEASE UPDATE: 19.14.0.0.220118 (33561310)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33561310/24538862/33561310_apply
_DBITST_CDBROOT_2022Mar01_11_15_43.log
	Status : SUCCESS

Patch Id : 33515361
	Action : APPLY
	Action Time : 01-MAR-2022 11:17:16
	Description : Database Release Update : 19.14.0.0.220118 (33515361)
	Logfile :
/u01/app/odaorabase/oracle/cfgtoollogs/sqlpatch/33515361/24589353/33515361_apply
_DBITST_CDBROOT_2022Mar01_11_15_43.log
	Status : SUCCESS

PL/SQL procedure successfully completed.
exit

Final checks

Let’s get the final versions:

odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
dbi-oda-x8
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB
                                          19.14.0.0.220118      up-to-date
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date
ILOM
                                          5.0.2.24.r141466      up-to-date
BIOS
                                          52050300              up-to-date
SHARED CONTROLLER FIRMWARE
                                          VDV1RL04              up-to-date
LOCAL DISK FIRMWARE
                                          1132                  up-to-date
SHARED DISK FIRMWARE
                                          1132                  up-to-date
HMP
                                          2.4.8.0.600           up-to-date

Everything is fine.

Cleanse the old patches

The old patches will never be used again, so don’t forget to remove previous patch files from the repository if your ODA has already been patched:

odacli cleanup-patchrepo -cl -comp db,gi -v 19.13.0.0.0

Put back your own settings

    Once everything is OK, don’t forget to put back your settings:

  • add your additional rpms manually if needed
  • put back your profile scripts for grid and oracle users

Patching an existing DB System

If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System.

odacli list-dbsystems
Name                  Shape       Cores  Memory      GI version          DB version          Status           Created                  Updated
--------------------  ----------  -----  ----------  ------------------  ------------------  ---------------  -----------------------  -----------------------
srvdb47               odb2        2      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 17:40:42 CET  2022-02-22 18:13:44 CET
srvdb49               odb2        2      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 16:27:54 CET  2022-02-22 16:58:43 CET
srvdb48               odb2        2      16.00 GB    19.13.0.0.211019    19.13.0.0.211019    CONFIGURED       2022-02-22 15:53:25 CET  2022-02-22 16:23:22 CET

Let’s do a describe component in one of the DB systems:

ssh srvdb48
odacli describe-component | grep -v ^$
System Version
---------------
19.13.0.0.0
System node Name
---------------
srvdb48
Local System Version
---------------
19.13.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK                                       19.13.0.0.0           19.14.0.0.0
GI                                        19.13.0.0.211019      19.14.0.0.220118
DB                                        19.13.0.0.211019      19.14.0.0.220118
DCSCONTROLLER                             19.13.0.0.0           19.14.0.0.0
DCSCLI                                    19.13.0.0.0           19.14.0.0.0
DCSAGENT                                  19.13.0.0.0           19.14.0.0.0
DCSADMIN                                  19.13.0.0.0           19.14.0.0.0
OS                                        7.9                   up-to-date

Patching method is the same as bare metal:

odacli update-dcsadmin -v 19.14.0.0.0
sleep 30; odacli update-dcscomponents -v 19.14.0.0.0
odacli update-dcsagent -v 19.14.0.0.0
sleep 180; odacli create-prepatchreport -s -v 19.14.0.0.0
odacli describe-prepatchreport -i 605cb646-9d39-4db4-bdcf-aca7f813942d


Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  605cb646-9d39-4db4-bdcf-aca7f813942d
            Description:  Patch pre-checks for [OS, GI, ORACHKSERVER]
                 Status:  FAILED
                Created:  March 1, 2022 11:48:18 AM CET
                 Result:  One or more pre-checks failed for [GI, ORACHK]

Node Name
---------------
srvdb48

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate BM versions            Failed
Validate kernel log level       Aborted   Internal error encountered
Validate minimum agent version  Aborted   Internal error encountered
Validate Central Inventory      Aborted   Internal error encountered
Validate patching locks         Aborted   Internal error encountered
Validate clones location exist  Aborted   Internal error encountered
Evaluate GI patching            Aborted   Internal error encountered
Validate command execution      Aborted   Internal error encountered

__ORACHK__
Running orachk                  Aborted   Internal error encountered
Validate command execution      Aborted   Internal error encountered

There is a bug with the repository mounted from the bare metal server, as described in the documentation:

cp /opt/oracle/oak/pkgrepos/System/VERSION /opt/oracle/oak/conf/VERSION
umount /opt/oracle/oak/pkgrepos
mount 192.168.17.2:/opt/oracle/oak/pkgrepos /opt/oracle/oak/pkgrepos

odacli create-prepatchreport -s -v 19.14.0.0.0
odacli describe-prepatchreport -i cfb24814-5c77-4ccb-9cee-eda971467a57

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  cfb24814-5c77-4ccb-9cee-eda971467a57
            Description:  Patch pre-checks for [OS, GI, ORACHKSERVER]
                 Status:  SUCCESS
                Created:  March 1, 2022 3:21:59 PM CET
                 Result:  All pre-checks succeeded

Node Name
---------------
srvdb48

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.14.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate BM versions            Success   Validated BM server components
                                          versions
Validate kernel log level       Success   Successfully validated the OS log
                                          level
Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

odacli update-server -v 19.14.0.0.0
odacli create-prepatchreport -d -i d8e0d75d-a810-4b89-805e-a4de5d646ec4 -v 19.14.0.0.0
odacli describe-prepatchreport -i 15065354-6a5d-411e-ad69-5991fc560428
odacli update-dbhome -i d8e0d75d-a810-4b89-805e-a4de5d646ec4 -v 19.14.0.0.0
odacli delete-dbhome -i d8e0d75d-a810-4b89-805e-a4de5d646ec4
odacli describe-component | grep -v ^$
System Version
---------------
19.14.0.0.0
System node Name
---------------
srvdb48
Local System Version
---------------
19.14.0.0.0
Component                                Installed Version    Available Version
---------------------------------------- -------------------- --------------------
OAK
                                          19.14.0.0.0           up-to-date
GI
                                          19.14.0.0.220118      up-to-date
DB
                                          19.14.0.0.220118      up-to-date
DCSCONTROLLER
                                          19.14.0.0.0           up-to-date
DCSCLI
                                          19.14.0.0.0           up-to-date
DCSAGENT
                                          19.14.0.0.0           up-to-date
DCSADMIN
                                          19.14.0.0.0           up-to-date
OS
                                          7.9                   up-to-date

Remember that using multiple DB Systems is nice but it’s much more work when you need to patch.

Conclusion

This release doesn’t bring any new feature and thus is easy to apply if you come from 19.10 or later versions. So don’t hesitate because this patch packs the latest bug and security fixes.


Thumbnail [60x60]
by
Jérôme Dubar