Introduction
It’s been several weeks that patch 19.19 is available on ODA. And I finally had the possibility to test it.
What’s new?
This version brings latest PSUs to database and grid homes, with their bug fixes as usual. It also brings latest 21c and 19c DB Systems. You already know that 21c is an innovation release. 23c will be the next long-term release, but it’s not yet available. Oracle recently updated the support timeline accordingly, and 19c is now supported without any additional cost until the first quarter of 2026.
Not a lot of new features for this version, but some nice additions. Among them, the error correlation between odacli and the subsequent tools directly from the command line. It was reserved for the BUI in the first versions:
odacli describe-job -i 767bf32e-51fc-46e6-a314-85ce9c053da0
Job details
----------------------------------------------------------------
ID: 767bf32e-51fc-46e6-a314-85ce9c053da0
Description: Database service creation with db name: DBTEST
Status: Failure (To view Error Correlation report, run "odacli describe-job -i 767bf32e-51fc-46e6-a314-85ce9c053da0 --ecr" command)
Created: July 26, 2023 4:22:42 PM CEST
Message: DCS-10001:Internal error encountered: Insufficient disk space for dbShape odb1s.
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Database Service creation July 26, 2023 4:22:42 PM CEST July 26, 2023 4:22:43 PM CEST Failure
Database Service creation July 26, 2023 4:22:42 PM CEST July 26, 2023 4:22:43 PM CEST Failure
Validating the required disk space July 26, 2023 4:22:42 PM CEST July 26, 2023 4:22:43 PM CEST Failure
odacli describe-job -i 767bf32e-51fc-46e6-a314-85ce9c053da0 --ecr
...Which ODA is compatible with this 19.19 release?
The newest ODAs X9-2S/L/HA are supported, as well as X8, X7 and X6 series. X5-2HA is no longer supported starting from this release, and therefore becomes obsolete. I personally won’t regret it. On one hand, it brought high capacity storage for the ODA series, but on the other hand it was the less performant ODA of the familly with nearline SAS HDDs (7200 rpm) as a transition after 10K SAS HDDs and before SSDs/NVMes.
Is this patch a cumulative one?
This 19.19 can be applied on top of 19.15 or later. I just applied this new patch on an X6-2M running on 19.8 using 19.12 and 19.16 as intermediate versions.
Is there also a patch for my databases?
Only databases version 19c are supported, and this is OK because this is the only one you should use now. You may use Data Preserving Reprovisioning if you come from a very old version (<=18.8) and want to keep your databases using 11g or 18c for example. I wrote a blog post on this feature several months ago.
If you absolutely need an older release, it should be possible to register an old DB clone belonging to an older patch without any problem. Several weeks ago, I successfully deployed a 12.2 DB home from 19.14 on a new X9-2S running 19.18, but an older 11gR2 didn’t work. If you really need 11gR2, you may configure an application VM on your ODA for that purpose, as described here, but it comes without any guarantee of any kind.
Download the patch and clone files
Download the patch and the corresponding clones to be able to apply the complete patch.
35290009 => the patch itself
30403673 => the GI clone needed for deploying newer GI version (mandatory)
30403662 => the DB clone for deploying new databases version of 19c
You don’t need the ISO file for patching, but I would recommend downloading it (patch 30403643).
Be sure to choose the very latest 19.19 when downloading the clones, download link from MOS will first propose older versions.
Prepare the patching
Before starting, please check these prerequisites:
- filesystems have 20% available free space (does not concern acfs volumes)
- additional rpms manually installed must be removed
- revert profile scripts to default’s one (for grid and oracle users)
- make sure you’ve planned a generous downtime, 4 hours being the bare minimum for patching and troubleshooting. 1 day is never too much. If you need to apply intermediate patches, you will need 4 hours for each jump
- you may need to remove old kernel version if you already patched multiple time your ODA, I described this in the next chapter
- I recently included a sanity reboot before patching in my procedure: I strongly recommend it
You should (must) use ODABR to make snapshots of the important filesystems prior patching. It’s fast and it doesn’t cost anything. I would also recommend doing a backup of most important files: tnsnames.ora and listener.ora, database list, network configuration files, aso.
Clean-up old kernels
If you have already patched multiple times your ODA, you may need to clean-up old kernels. Basically, each time an OS patch is applied, a new kernel is installed and it may filled up your /boot volume:
df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/md126p2 477M 359M 93M 80% /boot
uname -a
Linux chlaora01 4.14.35-2047.510.5.4.el7uek.x86_64 #2 SMP Thu Jan 20 09:10:35 PST 2022 x86_64 x86_64 x86_64 GNU/Linux
rpm -qa | grep kernel-uek-4
kernel-uek-4.14.35-2047.510.5.4.el7uek.x86_64
kernel-uek-4.14.35-2025.404.1.2.el7uek.x86_64
kernel-uek-4.14.35-1902.301.1.el7uek.x86_64In this example, current kernel is 4.14.35-2047. The 1902 and 2025 kernel headers are not needed anymore. Let’s manually remove the packages:
rpm -e kernel-uek-4.14.35-1902.301.1.el7uek.x86_64
rpm -e kernel-uek-4.14.35-2025.404.1.2.el7uek.x86_64
df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/md126p2 477M 153M 300M 34% /bootFar better, isn’t it?
Version precheck
Start to check current version on all components:
odacli describe-component
System Version
---------------
19.16.0.0.0
System node Name
---------------
oda2
Local System Version
---------------
19.16.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.16.0.0.0 up-to-date
GI
19.16.0.0.220719 up-to-date
DB {
[OraDB19000_home1 [POSTDT1,PMDT1]]
19.8.0.0.200714 19.16.0.0.220719
[OraDB11204_home1 [CORIDT2,CORIDV2,
IASDT2,IASIT1,ACTEDT1,SMTEST,IASAT1,
GAIATEST]] 11.2.0.4.200714 not-available
[OraDB12102_home1 [HORUSDT1,ATALDT2,
DELOSDT1,DELOSDT2,SOLISDT1,SOLISDT2]] 12.1.0.2.200114 12.1.0.2.220719
}
DCSCONTROLLER
19.16.0.0.0 up-to-date
DCSCLI
19.16.0.0.0 up-to-date
DCSAGENT
19.16.0.0.0 up-to-date
DCSADMIN
19.16.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.1.0.20.r145377 up-to-date
BIOS
38360300 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c2]
4.650.00-7176 up-to-date
[c3]
000161AB up-to-date
[c4]
800005DD up-to-date
[c5]
800005DE up-to-date
}
SHARED CONTROLLER FIRMWARE
KPYAJR3Q up-to-date
LOCAL DISK FIRMWARE
GXM10R3Q up-to-date
HMP
2.4.8.9.601 up-to-dateOnce the patch will be registered in the ODA repository, the “Available Version” column will be updated with versions provided within the patch.
Prepararing the patch and backing up the system
Copy the patch files to your ODA in a temp directory. On ODA X9-2, system disks are now smaller, so don’t hesitate to put the file on a nfs share or in the FRA acfs volume if you have acfs configured for your databases. Then unzip the files:
cd /opt/oren
unzip p30403673_1916000_Linux-x86-64.zip
unzip p34376263_1916000_Linux-x86-64.zip
unzip p30403662_1919000_Linux-x86-64.zip
rm p30403673_1916000_Linux-x86-64.zip p34376263_1916000_Linux-x86-64.zip p30403662_1919000_Linux-x86-64.zipIf not yet done, install the latest ODABR from MOS note 2466177.1:
rpm -ivh odabr-2.0.1-72.noarch.rpm
Check free space in the physical volume and choose adequate snapshot size for each of the 3 filesystems. These settings were OK for me:
pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 VolGroupSys lvm2 a-- <439.44g <123.44g
/opt/odabr/odabr backup -snap -osize 55 -rsize 15 -usize 40
INFO: 2023-07-26 14:14:48: Please check the logfile '/opt/odabr/out/log/odabr_13988.log' for more details
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
odabr - ODA node Backup Restore - Version: 2.0.1-72
Copyright 2013, 2023, Oracle and/or its affiliates.
--------------------------------------------------------
Author: Ruggero Citton <[email protected]>
RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
INFO: 2023-07-26 14:14:48: Checking superuser
INFO: 2023-07-26 14:14:48: Checking Bare Metal
INFO: 2023-07-26 14:14:48: Removing existing LVM snapshots
WARNING: 2023-07-26 14:14:48: LVM snapshot for 'opt' does not exist
WARNING: 2023-07-26 14:14:48: LVM snapshot for 'u01' does not exist
WARNING: 2023-07-26 14:14:48: LVM snapshot for 'root' does not exist
INFO: 2023-07-26 14:14:48: Checking current OS version
INFO: 2023-07-26 14:14:48: Checking LVM restore backgroud process
INFO: 2023-07-26 14:14:48: Checking LVM size
INFO: 2023-07-26 14:14:48: Boot device backup
INFO: 2023-07-26 14:14:48: Getting EFI device
WARNING: 2023-07-26 14:14:48: Failing checking efi device
INFO: 2023-07-26 14:14:48: ...no EFI device found
INFO: 2023-07-26 14:14:48: Getting boot device
INFO: 2023-07-26 14:14:48: ...step1 - making boot device backup using tar
SUCCESS: 2023-07-26 14:15:06: ...boot content saved as '/opt/odabr/out/hbi/boot.tar.gz'
INFO: 2023-07-26 14:15:06: ...step2 - unmounting boot
INFO: 2023-07-26 14:15:06: ...step3 - making boot device backup using dd
SUCCESS: 2023-07-26 14:15:09: ...boot device backup saved as '/opt/odabr/out/hbi/boot.img'
INFO: 2023-07-26 14:15:09: ...step4 - mounting boot
INFO: 2023-07-26 14:15:09: ...step5 - checking boot device backup
INFO: 2023-07-26 14:15:10: Making OCR physical backup
INFO: 2023-07-26 14:15:15: ...ocr backup saved as '/opt/odabr/out/hbi/ocrbackup_13988.bck'
SUCCESS: 2023-07-26 14:15:15: OCR physical backup created successfully
INFO: 2023-07-26 14:15:15: OCR export backup
INFO: 2023-07-26 14:15:17: ...ocr export saved as '/opt/odabr/out/hbi/ocrexport_13988.bck'
SUCCESS: 2023-07-26 14:15:17: OCR export backup created successfully
INFO: 2023-07-26 14:15:17: Making LVM snapshot backup
SUCCESS: 2023-07-26 14:15:17: ...snapshot backup for 'opt' created successfully
SUCCESS: 2023-07-26 14:15:18: ...snapshot backup for 'u01' created successfully
SUCCESS: 2023-07-26 14:15:18: ...snapshot backup for 'root' created successfully
SUCCESS: 2023-07-26 14:15:18: LVM snapshots backup done successfully
/opt/odabr/odabr infosnap
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
odabr - ODA node Backup Restore - Version: 2.0.1-72
Copyright 2013, 2023, Oracle and/or its affiliates.
--------------------------------------------------------
Author: Ruggero Citton <[email protected]>
RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
LVM snap name Status COW Size Data%
------------- ---------- ---------- ------
root_snap active 15.00 GiB 0.01%
opt_snap active 55.00 GiB 0.01%
u01_snap active 40.00 GiB 0.02%
If you need to apply multiple patches, I would recommend erasing the snapshots after each patch successfully applied, and creating new ones once patch files are unzipped.
Updating the DCS tools
odacli update-repository -f /opt/oren/oda-sm-19.19.0.0.0-230510-server.zip
sleep 60 ; odacli describe-job -i "e0c8d130-a162-4364-a942-e4d4e89f4bdf"
Job details
----------------------------------------------------------------
ID: e0c8d130-a162-4364-a942-e4d4e89f4bdf
Description: Repository Update
Status: Success
Created: July 26, 2023 2:16:43 PM CEST
Message: /opt/oren/oda-sm-19.19.0.0.0-230510-server.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle July 26, 2023 2:16:44 PM CEST July 26, 2023 2:17:19 PM CEST Success
odacli describe-component
System Version
---------------
19.16.0.0.0
System node Name
---------------
oda2
Local System Version
---------------
19.16.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.16.0.0.0 19.19.0.0.0
GI
19.16.0.0.220719 19.19.0.0.230418
DB {
[OraDB19000_home1 [POSTDT1,PMDT1]]
19.8.0.0.200714 19.19.0.0.230418
[OraDB11204_home1 [CORIDT2,CORIDV2,
IASDT2,IASIT1,ACTEDT1,SMTEST,IASAT1,
GAIATEST]] 11.2.0.4.200714 not-available
[OraDB12102_home1 [HORUSDT1,ATALDT2,
DELOSDT1,DELOSDT2,SOLISDT1,SOLISDT2]] 12.1.0.2.200114 not-available
}
DCSCONTROLLER
19.16.0.0.0 19.19.0.0.0
DCSCLI
19.16.0.0.0 19.19.0.0.0
DCSAGENT
19.16.0.0.0 19.19.0.0.0
DCSADMIN
19.16.0.0.0 19.19.0.0.0
OS
7.9 up-to-date
ILOM
5.1.0.20.r145377 5.1.1.21.r149558
BIOS
38360300 38370200
LOCAL CONTROLLER FIRMWARE {
[c2]
4.650.00-7176 up-to-date
[c3]
000161AB up-to-date
[c4]
800005DD up-to-date
[c5]
800005DE up-to-date
}
SHARED CONTROLLER FIRMWARE
KPYAJR3Q up-to-date
LOCAL DISK FIRMWARE
GXM10R3Q up-to-date
HMP
2.4.8.9.601 2.4.8.9.605
Let’s start with the DCS components update:
odacli update-dcsadmin -v 19.19.0.0.0
odacli describe-job -i "21d32707-245b-4712-b683-41e1274fb420"
Job details
----------------------------------------------------------------
ID: 21d32707-245b-4712-b683-41e1274fb420
Description: DcsAdmin patching
Status: Success
Created: July 26, 2023 2:21:12 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation July 26, 2023 2:21:13 PM CEST July 26, 2023 2:21:13 PM CEST Success
Dcs-admin upgrade July 26, 2023 2:21:13 PM CEST July 26, 2023 2:21:19 PM CEST Success
odacli update-dcscomponents -v 19.19.0.0.0
{
"jobId" : "51b60358-de51-4c7f-b68c-a346e5e80099",
"status" : "Success",
"message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful. dcscli RPM upgrade is successful. dcscontroller RPM upgrade is successful. Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful. ",
"reports" : null,
"createTimestamp" : "July 26, 2023 14:22:33 PM CEST",
"description" : "Update-dcscomponents job completed and is not part of Agent job list",
"updatedTime" : "July 26, 2023 14:24:20 PM CEST"
}
sleep 20 ; odacli update-dcsagent -v 19.19.0.0.0
odacli describe-job -i "2c03c606-405e-4763-b31a-34149d22f0f2"
Job details
----------------------------------------------------------------
ID: 2c03c606-405e-4763-b31a-34149d22f0f2
Description: DcsAgent patching
Status: Success
Created: July 26, 2023 2:24:44 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Dcs-agent upgrade to version July 26, 2023 2:24:44 PM CEST July 26, 2023 2:26:38 PM CEST Success
19.19.0.0.0
Update System version July 26, 2023 2:26:39 PM CEST July 26, 2023 2:26:39 PM CEST SuccessRegister the 19.19 GI clone in the ODA repository:
odacli update-repository -f /opt/oren/odacli-dcs-19.19.0.0.0-230510-GI-19.19.0.0.zip
sleep 45; odacli describe-job -i "1763af8a-1407-4ed4-af4f-462761a4eb1d"
Job details
----------------------------------------------------------------
ID: 1763af8a-1407-4ed4-af4f-462761a4eb1d
Description: Repository Update
Status: Success
Created: July 26, 2023 2:29:06 PM CEST
Message: /opt/oren/odacli-dcs-19.19.0.0.0-230510-GI-19.19.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle July 26, 2023 2:29:07 PM CEST July 26, 2023 2:29:54 PM CEST Success
System prepatching report
Let’s do the prepatching test:
odacli create-prepatchreport -s -v 19.19.0.0.0
sleep 360; odacli describe-prepatchreport -i e99c3848-0ccb-4085-a15d-398f7ec91d59
Patch pre-check report
------------------------------------------------------------------------
Job ID: e99c3848-0ccb-4085-a15d-398f7ec91d59
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
Status: FAILED
Created: July 26, 2023 2:30:58 PM CEST
Result: One or more pre-checks failed for [ORACHK]
Node Name
---------------
oda2
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.19.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.19.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate GI metadata Success Successfully validated GI metadata
Validate supported GI versions Success Successfully validated minimum version
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.19.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate kernel log level Success Successfully validated the OS log
level
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate Central Inventory Success oraInventory validation passed
Validate patching locks Success Validated patching locks
Validate clones location exist Success Validated clones location
Validate DB start dependencies Success DBs START dependency check passed
Validate DB stop dependencies Success DBs STOP dependency check passed
Validate space for clones Success Clones volume is already created
volume
Evaluate GI patching Success Successfully validated GI patching
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Failed failed to run ORAchk: Failed to run
orachk
Validate command execution Success Validated command execution
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Validated command executionEverything is OK apart from ORACHK. But this shouldn’t be a problem, it’s about patching the system now. If a failure is detected on the other checks, I definitly won’t recommend going further.
Patching server and GI
Let’s start the update-server, force option is mandatory because of the failure of ORACHK:
odacli update-server -f -v 19.19.0.0.0
odacli describe-job -i 02430d8e-6dff-4bcc-91f8-c7f20fcb9ad2
Job details
----------------------------------------------------------------
ID: 02430d8e-6dff-4bcc-91f8-c7f20fcb9ad2
Description: Server Patching
Status: Success
Created: July 26, 2023 2:41:36 PM CEST
Message: Successfully patched GI with RHP
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata July 26, 2023 2:41:55 PM CEST July 26, 2023 2:41:55 PM CEST Success
Validate ILOM server reachable July 26, 2023 2:41:55 PM CEST July 26, 2023 2:41:56 PM CEST Success
Validate DCS Admin mTLS setup July 26, 2023 2:41:56 PM CEST July 26, 2023 2:41:56 PM CEST Success
Configure export clones resource July 26, 2023 2:41:57 PM CEST July 26, 2023 2:41:58 PM CEST Success
Creating repositories using yum July 26, 2023 2:41:58 PM CEST July 26, 2023 2:42:02 PM CEST Success
Updating YumPluginVersionLock rpm July 26, 2023 2:42:02 PM CEST July 26, 2023 2:42:02 PM CEST Success
Applying OS Patches July 26, 2023 2:42:02 PM CEST July 26, 2023 2:56:06 PM CEST Success
Creating repositories using yum July 26, 2023 2:56:07 PM CEST July 26, 2023 2:56:07 PM CEST Success
Applying HMP Patches July 26, 2023 2:56:07 PM CEST July 26, 2023 2:56:30 PM CEST Success
Patch location validation July 26, 2023 2:56:30 PM CEST July 26, 2023 2:56:30 PM CEST Success
Oda-hw-mgmt upgrade July 26, 2023 2:56:31 PM CEST July 26, 2023 2:57:07 PM CEST Success
OSS Patching July 26, 2023 2:57:07 PM CEST July 26, 2023 2:57:08 PM CEST Success
Applying Firmware Disk Patches July 26, 2023 2:57:08 PM CEST July 26, 2023 2:57:20 PM CEST Success
Applying Firmware Controller Patches July 26, 2023 2:57:20 PM CEST July 26, 2023 2:57:31 PM CEST Success
Checking Ilom patch Version July 26, 2023 2:57:31 PM CEST July 26, 2023 2:57:32 PM CEST Success
Patch location validation July 26, 2023 2:57:32 PM CEST July 26, 2023 2:57:32 PM CEST Success
Save password in Wallet July 26, 2023 2:57:32 PM CEST July 26, 2023 2:57:33 PM CEST Success
Disabling IPMI v2 July 26, 2023 2:57:33 PM CEST July 26, 2023 2:57:36 PM CEST Success
Apply Ilom patch July 26, 2023 2:57:36 PM CEST July 26, 2023 3:14:07 PM CEST Success
Copying Flash Bios to Temp location July 26, 2023 3:14:07 PM CEST July 26, 2023 3:14:07 PM CEST Success
Starting the clusterware July 26, 2023 3:14:07 PM CEST July 26, 2023 3:15:47 PM CEST Success
Registering image July 26, 2023 3:15:47 PM CEST July 26, 2023 3:15:47 PM CEST Success
Registering working copy July 26, 2023 3:15:47 PM CEST July 26, 2023 3:15:48 PM CEST Success
Registering image July 26, 2023 3:15:48 PM CEST July 26, 2023 3:15:48 PM CEST Success
Creating GI home directories July 26, 2023 3:15:48 PM CEST July 26, 2023 3:15:48 PM CEST Success
Extract GI clone July 26, 2023 3:15:48 PM CEST July 26, 2023 3:15:48 PM CEST Success
Provisioning Software Only GI with RHP July 26, 2023 3:15:48 PM CEST July 26, 2023 3:15:48 PM CEST Success
Patch GI with RHP July 26, 2023 3:15:48 PM CEST July 26, 2023 3:22:10 PM CEST Success
Updating .bashrc July 26, 2023 3:22:10 PM CEST July 26, 2023 3:22:10 PM CEST Success
Updating GIHome in the repository July 26, 2023 3:22:10 PM CEST July 26, 2023 3:22:10 PM CEST Success
Updating GIHome version July 26, 2023 3:22:10 PM CEST July 26, 2023 3:22:15 PM CEST Success
Validate GI availability July 26, 2023 3:22:26 PM CEST July 26, 2023 3:22:27 PM CEST Success
Patch KVM CRS type July 26, 2023 3:22:27 PM CEST July 26, 2023 3:22:27 PM CEST Success
Patch VM vDisks CRS dependencies July 26, 2023 3:22:27 PM CEST July 26, 2023 3:22:27 PM CEST Success
Patch DB System domain config July 26, 2023 3:22:27 PM CEST July 26, 2023 3:22:27 PM CEST Success
Update System version July 26, 2023 3:23:09 PM CEST July 26, 2023 3:23:09 PM CEST Success
Cleanup JRE Home July 26, 2023 3:23:09 PM CEST July 26, 2023 3:23:10 PM CEST Success
Add SYSNAME in Env July 26, 2023 3:23:10 PM CEST July 26, 2023 3:23:10 PM CEST Success
Starting the clusterware July 26, 2023 3:23:10 PM CEST July 26, 2023 3:24:48 PM CEST Success
Setting ACL for disk groups July 26, 2023 3:24:48 PM CEST July 26, 2023 3:24:54 PM CEST Success
Enable LKCE July 26, 2023 3:27:02 PM CEST July 26, 2023 3:31:01 PM CEST Success
Update previous workarounds July 26, 2023 3:31:09 PM CEST July 26, 2023 3:31:09 PM CEST Success
Generating and saving BOM July 26, 2023 3:31:09 PM CEST July 26, 2023 3:33:54 PM CEST Success
PreRebootNode Actions July 26, 2023 3:33:54 PM CEST July 26, 2023 3:34:39 PM CEST Success
Reboot Ilom July 26, 2023 3:34:39 PM CEST July 26, 2023 3:34:39 PM CEST SuccessServer reboots several minutes after the patch ends. On this X6-2M, server patching took 55 minutes.
odacli describe-component
System Version
---------------
19.19.0.0.0
System node Name
---------------
oda2
Local System Version
---------------
19.19.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.19.0.0.0 up-to-date
GI
19.19.0.0.230418 up-to-date
DB {
[OraDB19000_home1 [POSTDT1,PMDT1]]
19.8.0.0.200714 19.19.0.0.230418
[OraDB11204_home1 [CORIDT2,CORIDV2,
IASDT2,IASIT1,ACTEDT1,SMTEST,IASAT1,
GAIATEST]] 11.2.0.4.200714 not-available
[OraDB12102_home1 [HORUSDT1,ATALDT2,
DELOSDT1,DELOSDT2,SOLISDT1,SOLISDT2]] 12.1.0.2.200114 not-available
}
DCSCONTROLLER
19.19.0.0.0 up-to-date
DCSCLI
19.19.0.0.0 up-to-date
DCSAGENT
19.19.0.0.0 up-to-date
DCSADMIN
19.19.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.1.1.21.r149558 up-to-date
BIOS
38370200 up-to-date
LOCAL CONTROLLER FIRMWARE {
[c2]
4.650.00-7176 up-to-date
[c3]
000161AB up-to-date
[c4]
800005DD up-to-date
[c5]
800005DE up-to-date
}
SHARED CONTROLLER FIRMWARE
KPYAJR3Q up-to-date
LOCAL DISK FIRMWARE
GXM10R3Q up-to-date
HMP
2.4.8.9.605 up-to-datePatching the storage
Patching the storage is only needed if describe-component tells you that you’re not up-to-date. On my X6-2M it wasn’t needed, but on your ODA if you need to, it’s quite straight forward:
odacli update-storage -v 19.19.0.0.0
odacli describe-job -i ...The server will also reboot when it’s done.
Patching the DB homes
Time for patching the DB homes depends on the number of DB homes and number of databases you have. In this other example (X8-2M), I will apply the patch on my 19.12 DB home. For DB homes, you don’t need to use intermediate patches (and you will save time).
odacli update-repository -f /opt/oren/odacli-dcs-19.19.0.0.0-230510-DB-19.19.0.0.zip
sleep 30; odacli describe-job -i e078e543-10c0-438b-8cf4-ec24de7926e9
Job details
----------------------------------------------------------------
ID: e078e543-10c0-438b-8cf4-ec24de7926e9
Description: Repository Update
Status: Success
Created: August 11, 2023 9:50:42 AM CEST
Message: /opt/oren/odacli-dcs-19.19.0.0.0-230510-DB-19.19.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle August 11, 2023 9:50:42 AM CEST August 11, 2023 9:51:21 AM CEST Success
odacli list-dbhomes
ID Name DB Version DB Edition Home Location Status
---------------------------------------- -------------------- ---------------------------------------- ---------- ------------------------------------------------------- ----------
5d04a23f-f282-43fb-b95c-3a4d93d007e7 OraDB19000_home1 19.12.0.0.210720 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
689f3b27-e122-4f70-825f-6221361ff3db OraDB19000_home2 19.12.0.0.210720 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED
Let's patch the first DB home here:
odacli create-prepatchreport -d -i 5d04a23f-f282-43fb-b95c-3a4d93d007e7 -v 19.19.0.0.0
sleep 600; odacli describe-prepatchreport -i 621e5e56-744b-4c82-85cb-6f86a0204810
Patch pre-check report
------------------------------------------------------------------------
Job ID: 621e5e56-744b-4c82-85cb-6f86a0204810
Description: Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
Status: SUCCESS
Created: August 11, 2023 10:07:07 AM CEST
Result: All pre-checks succeeded
Node Name
---------------
oda-prod-01
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__DB__
Validate DB Home ID Success Validated DB Home ID:
5d04a23f-f282-43fb-b95c-3a4d93d007e7
Validate patching tag Success Validated patching tag: 19.19.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate minimum agent version Success Validated minimum agent version
Is GI upgraded Success Validated GI is upgraded
Validate available space for Success Validated free space required under
db /u01
Validate dbHomesOnACFS Success User has configured disk group for
configured Database homes on ACFS
Validate Oracle base Success Successfully validated Oracle Base
Is DB clone available Success Successfully validated clone file
exists
Evaluate DBHome patching with Success Successfully validated updating
RHP dbhome with RHP. and local patching
is possible
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Successfully ran Orachk
Validate command execution Success Validated command executionSometimes ORACHK will warn you about different things, but your database may need some specific settings ORACHK won’t like, so you could also force the DB home update with -f. But only these warnings from ORACHK please.
odacli update-dbhome -i 5d04a23f-f282-43fb-b95c-3a4d93d007e7 -v 19.19.0.0.0
sleep 600; odacli describe-job -i cca4b034-aa11-4250-946f-b7d28c794fc0
Job details
----------------------------------------------------------------
ID: cca4b034-aa11-4250-946f-b7d28c794fc0
Description: DB Home Patching: Home Id is 5d04a23f-f282-43fb-b95c-3a4d93d007e7
Status: Success
Created: August 11, 2023 1:09:56 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating wallet for DB Client August 11, 2023 1:10:10 PM CEST August 11, 2023 1:10:10 PM CEST Success
Patch databases by RHP - [REPFMW] August 11, 2023 1:10:10 PM CEST August 11, 2023 1:16:08 PM CEST Success
Updating database metadata August 11, 2023 1:16:08 PM CEST August 11, 2023 1:16:08 PM CEST Success
Set log_archive_dest for Database August 11, 2023 1:16:08 PM CEST August 11, 2023 1:16:11 PM CEST Success
Patch databases by RHP - [PAMIF] August 11, 2023 1:16:11 PM CEST August 11, 2023 1:21:48 PM CEST Success
Updating database metadata August 11, 2023 1:21:48 PM CEST August 11, 2023 1:21:48 PM CEST Success
Set log_archive_dest for Database August 11, 2023 1:21:48 PM CEST August 11, 2023 1:21:51 PM CEST Success
Update System version August 11, 2023 1:21:51 PM CEST August 11, 2023 1:21:51 PM CEST Success
Generating and saving BOM August 11, 2023 1:21:51 PM CEST August 11, 2023 1:22:49 PM CEST Success
TDE parameter update August 11, 2023 1:22:49 PM CEST August 11, 2023 1:22:49 PM CEST Success
odacli list-dbhomes
ID Name DB Version DB Edition Home Location Status
---------------------------------------- -------------------- ---------------------------------------- ---------- ------------------------------------------------------- ----------
5d04a23f-f282-43fb-b95c-3a4d93d007e7 OraDB19000_home1 19.12.0.0.210720 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
689f3b27-e122-4f70-825f-6221361ff3db OraDB19000_home2 19.12.0.0.210720 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED
83bfc182-bae6-4d8b-9491-aa2f2758b808 OraDB19000_home3 19.19.0.0.230418 EE /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_3 CONFIGUREDDon’t forget that the previous DB home will not be deleted: you should delete it after the patch of several days later.
Final checks
I would recommend checking the final versions:
odacli describe-component
...Cleanse the old patches
The old patches will never be used again, so don’t forget to remove them from the repository if your ODA has already been patched:
odacli cleanup-patchrepo -cl -v 19.16.0.0.0
odacli describe-job -i "ada6139b-4406-4e86-b978-73f7d5133385"
Job details
----------------------------------------------------------------
ID: ada6139b-4406-4e86-b978-73f7d5133385
Description: Cleanup patchrepos
Status: Success
Created: July 26, 2023 3:48:37 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Cleanup Repository July 26, 2023 3:48:37 PM CEST July 26, 2023 3:48:37 PM CEST Success
Cleanup JRE Home July 26, 2023 3:48:37 PM CEST July 26, 2023 3:48:37 PM CEST Success
Cleanup old ASR rpm July 26, 2023 3:48:37 PM CEST July 26, 2023 3:48:37 PM CEST SuccessPost-patching tasks
Once everything is OK, don’t forget to:
- add your additional rpms manually if needed
- put back your profile scripts for grid and oracle users
- remove the old DB homes if they are not needed anymore
odacli delete-dbhome -i 5d04a23f-f282-43fb-b95c-3a4d93d007e7
sleep 60 ; odacli describe-job -i odacli describe-job -i 8c79189a-dff6-41fe-b9a5-144a2bac1db1
Job details
----------------------------------------------------------------
ID: 8c79189a-dff6-41fe-b9a5-144a2bac1db1
Description: Database Home OraDB19000_home1 Deletion with id 5d04a23f-f282-43fb-b95c-3a4d93d007e7
Status: Success
Created: August 11, 2023 2:36:33 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate dbhome August 11, 2023 2:36:33 PM CEST August 11, 2023 2:36:33 PM CEST Success
5d04a23f-f282-43fb-b95c-3a4d93d007e7
for deletion
Setting up ssh equivalance August 11, 2023 2:36:33 PM CEST August 11, 2023 2:36:33 PM CEST Success
Setting up ssh equivalance August 11, 2023 2:36:33 PM CEST August 11, 2023 2:36:33 PM CEST Success
Deleting DbHome by RHP August 11, 2023 2:36:34 PM CEST August 11, 2023 2:37:09 PM CEST SuccessIf everything is fine, remove the snapshots done with ODABR:
/opt/odabr/odabr delsnap
INFO: 2023-07-26 15:51:19: Please check the logfile '/opt/odabr/out/log/odabr_71724.log' for more details
INFO: 2023-07-26 15:51:19: Removing LVM snapshots
INFO: 2023-07-26 15:51:19: ...removing LVM snapshot for 'opt'
SUCCESS: 2023-07-26 15:51:20: ...snapshot for 'opt' removed successfully
INFO: 2023-07-26 15:51:20: ...removing LVM snapshot for 'u01'
SUCCESS: 2023-07-26 15:51:20: ...snapshot for 'u01' removed successfully
INFO: 2023-07-26 15:51:20: ...removing LVM snapshot for 'root'
SUCCESS: 2023-07-26 15:51:20: ...snapshot for 'root' removed successfully
SUCCESS: 2023-07-26 15:51:20: Remove LVM snapshots done successfully
Patching DB Systems
If you use DB Systems on your ODA, meaning that some of your databases are running in dedicated VMs, you will need to apply the patch inside each DB System. I haven’t tried on my X6-2M because it only runs bare metal databases, but you will need to list your DB Systems, connect to each one, and apply the server and database patches:
odacli list-dbsystems
ssh ...
odacli update-server -v 19.19.0.0.0
odacli update-dbhome -i ... -v 19.19.0.0.0
Conclusion
This release is easy to apply coming from 19.16 and probably from newer versions. Next versions might bring major OS upgrade and 23c GI and DB homes.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/JDU_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/ALN_web-min-scaled.jpg)
ArthurGE
20.01.2024Wow, the guide works very well.
But , I think it will no longer works for version 19.21.
For example, updating an ODA HA 19.18 with large databases to 19.21