By Mouhamadou Diaw
Patching an ODA is always exciting. Preparing a patch of an ODA HA may seem more complicated that an single ODA. But it’s the same processus. All patching operations should be run on one node.
In this document I am describing the patch of a ODA HA from 19.14 to 19.16 with a customer. Great thanks to Pierre.
The documentation is well done. Just read and followw the steps
We downloaded following patches
Oracle Database Appliance Server Patch for the ODACLI/DCS stack : p34376263_1916000_Linux-x86-64.zip
Oracle Grid Infrastructure clone files : p30403673_1916000_Linux-x86-64.zip
Oracle Database clone files : p30403662_1916000_Linux-x86-64.zip
Once unpacked we will have following files
oda-sm-19.16.0.0.0-220809-server.zip
odacli-dcs-19.16.0.0.0-220717-GI-19.16.0.0.zip
odacli-dcs-19.16.0.0.0-220717-DB-19.16.0.0.zip
These files will be used to update the repository. They should be copied in the ODA server.
Note that all operations will be done only on a node. If some tasks need to be done on both nodes, we will specify
On one node of the ODA, connect as root and update the repository with the patch for the ODACLI/DCS stack
[root@oda-deva]# /opt/oracle/dcs/bin/odacli update-repository -f /opt/Downloads/oda-sm-19.16.0.0.0-220809-server.zip
Validate that the job was successful
[2022-10-12 11:43:04 root@oda-deva]# odacli describe-job -i "7a58f486-6ab1-4b4f-b9a6-a38458535d2c"
Job details
----------------------------------------------------------------
ID: 7a58f486-6ab1-4b4f-b9a6-a38458535d2c
Description: Repository Update
Status: Success
Created: October 12, 2022 11:42:25 AM CEST
Message: /opt/Downloads/oda-sm-19.16.0.0.0-220809-server.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace October 12, 2022 11:42:32 AM CEST October 12, 2022 11:42:32 AM CEST Success
Setting up ssh equivalance October 12, 2022 11:42:32 AM CEST October 12, 2022 11:42:32 AM CEST Success
Copy BundleFile October 12, 2022 11:42:33 AM CEST October 12, 2022 11:42:43 AM CEST Success
Validating CopiedFile October 12, 2022 11:42:43 AM CEST October 12, 2022 11:42:49 AM CEST Success
Unzip bundle October 12, 2022 11:42:49 AM CEST October 12, 2022 11:43:12 AM CEST Success
Unzip bundle October 12, 2022 11:43:18 AM CEST October 12, 2022 11:43:45 AM CEST Success
Removing ssh keys October 12, 2022 11:43:45 AM CEST October 12, 2022 11:43:46 AM CEST Success
Then update the DCS admin with user root
# /opt/oracle/dcs/bin/odacli update-dcsadmin -v 19.16.0.0.0
Validate the status of the job
[2022-10-12 11:45:57 root@oda-deva]# odacli describe-job -i "8d792424-e5c0-4ec5-8133-606a6d3632cb"
Job details
----------------------------------------------------------------
ID: 8d792424-e5c0-4ec5-8133-606a6d3632cb
Description: DcsAdmin patching
Status: Success
Created: October 12, 2022 11:45:57 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation October 12, 2022 11:45:57 AM CEST October 12, 2022 11:45:57 AM CEST Success
Patch location validation October 12, 2022 11:45:57 AM CEST October 12, 2022 11:45:57 AM CEST Success
dcs-admin upgrade October 12, 2022 11:45:58 AM CEST October 12, 2022 11:46:04 AM CEST Success
dcs-admin upgrade October 12, 2022 11:46:04 AM CEST October 12, 2022 11:46:10 AM CEST Success
After update the DCS components with root
[2022-10-12 11:46:49 root@oda-deva]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.16.0.0.0
{
"jobId" : "2bc1aebd-50a7-41fe-b66e-b08ed0bc83d1",
"status" : "Success",
"message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent shutdown is successful. MySQL upgrade is successful. Metadata migration is done before. Metadata schema update is done. dcsagent RPM upgrade is successful. dcscli RPM upgrade is successful. dcscontroller RPM upgrade is successful. Successfully ran setupAgentAuth.sh zookeeper RPM upgrade is successful. ",
"reports" : null,
"createTimestamp" : "October 12, 2022 11:47:03 AM CEST",
"description" : "Update-dcscomponents job completed and is not part of Agent job list",
"updatedTime" : "October 12, 2022 11:50:34 AM CEST"
}
The next step is to update the DCS agent with the user root
# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.16.0.0.0
We still have to validate that the job was successfull
[2022-10-12 11:56:25 root@oda-deva]# odacli describe-job -i "350ebfca-0ed9-48da-8ed7-ba37dba1df44"
Job details
----------------------------------------------------------------
ID: 350ebfca-0ed9-48da-8ed7-ba37dba1df44
Description: DcsAgent patching
Status: Success
Created: October 12, 2022 11:51:53 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Stop DCS Admin October 12, 2022 11:51:59 AM CEST October 12, 2022 11:51:59 AM CEST Success
Stop DCS Admin October 12, 2022 11:51:59 AM CEST October 12, 2022 11:51:59 AM CEST Success
Generate mTLS certificates October 12, 2022 11:51:59 AM CEST October 12, 2022 11:52:00 AM CEST Success
Generate mTLS certificates October 12, 2022 11:51:59 AM CEST October 12, 2022 11:52:01 AM CEST Success
Exporting Public Keys October 12, 2022 11:52:01 AM CEST October 12, 2022 11:52:02 AM CEST Success
Exporting Public Keys October 12, 2022 11:52:01 AM CEST October 12, 2022 11:52:02 AM CEST Success
Copy Public Key to Remote node October 12, 2022 11:52:02 AM CEST October 12, 2022 11:52:03 AM CEST Success
Copy Public Key to Remote node October 12, 2022 11:52:03 AM CEST October 12, 2022 11:52:04 AM CEST Success
Creating Trust Store October 12, 2022 11:52:04 AM CEST October 12, 2022 11:52:06 AM CEST Success
Creating Trust Store October 12, 2022 11:52:04 AM CEST October 12, 2022 11:52:06 AM CEST Success
Update config files October 12, 2022 11:52:06 AM CEST October 12, 2022 11:52:06 AM CEST Success
Update config files October 12, 2022 11:52:06 AM CEST October 12, 2022 11:52:06 AM CEST Success
Restart DCS Admin October 12, 2022 11:52:06 AM CEST October 12, 2022 11:52:27 AM CEST Success
Restart DCS Admin October 12, 2022 11:52:06 AM CEST October 12, 2022 11:52:27 AM CEST Success
Dcs-agent upgrade to version 19.16.0.0.0 October 12, 2022 11:52:27 AM CEST October 12, 2022 11:54:27 AM CEST Success
Dcs-agent upgrade to version 19.16.0.0.0 October 12, 2022 11:54:27 AM CEST October 12, 2022 11:56:27 AM CEST Success
Update System version October 12, 2022 11:56:28 AM CEST October 12, 2022 11:56:28 AM CEST Success
Update System version October 12, 2022 11:56:28 AM CEST October 12, 2022 11:56:28 AM CEST Success
Now it’s time to update the repository with the GI clone we downloaded
# /opt/oracle/dcs/bin/odacli update-repository -f /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-GI-19.16.0.0.zip
And to validate the status of the job
[2022-10-12 12:03:17 root@oda-deva]# odacli describe-job -i "921447ac-be4c-470d-a683-bb4d5bc287ec"
Job details
----------------------------------------------------------------
ID: 921447ac-be4c-470d-a683-bb4d5bc287ec
Description: Repository Update
Status: Success
Created: October 12, 2022 12:00:37 PM CEST
Message: /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-GI-19.16.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace October 12, 2022 12:00:50 PM CEST October 12, 2022 12:00:50 PM CEST Success
Setting up ssh equivalance October 12, 2022 12:00:50 PM CEST October 12, 2022 12:00:53 PM CEST Success
Copy BundleFile October 12, 2022 12:00:53 PM CEST October 12, 2022 12:01:16 PM CEST Success
Validating CopiedFile October 12, 2022 12:01:16 PM CEST October 12, 2022 12:01:43 PM CEST Success
Unzip bundle October 12, 2022 12:01:49 PM CEST October 12, 2022 12:02:37 PM CEST Success
Unzip bundle October 12, 2022 12:02:37 PM CEST October 12, 2022 12:03:25 PM CEST Success
Delete PatchBundles October 12, 2022 12:03:25 PM CEST October 12, 2022 12:03:26 PM CEST Success
Update the repository with the DB clone
# /opt/oracle/dcs/bin/odacli update-repository -f /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-DB-19.16.0.0.zip
[2022-10-12 12:05:52 root@oda-deva]# odacli describe-job -i "ad8eed4c-7c2e-4e76-b984-e0e19467d909"
Job details
----------------------------------------------------------------
ID: ad8eed4c-7c2e-4e76-b984-e0e19467d909
Description: Repository Update
Status: Success
Created: October 12, 2022 12:04:03 PM CEST
Message: /opt/Downloads/odacli-dcs-19.16.0.0.0-220717-DB-19.16.0.0.zip
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Check AvailableSpace October 12, 2022 12:04:14 PM CEST October 12, 2022 12:04:14 PM CEST Success
Setting up ssh equivalance October 12, 2022 12:04:14 PM CEST October 12, 2022 12:04:15 PM CEST Success
Copy BundleFile October 12, 2022 12:04:15 PM CEST October 12, 2022 12:04:31 PM CEST Success
Validating CopiedFile October 12, 2022 12:04:31 PM CEST October 12, 2022 12:04:41 PM CEST Success
Unzip bundle October 12, 2022 12:04:41 PM CEST October 12, 2022 12:05:19 PM CEST Success
Unzip bundle October 12, 2022 12:05:19 PM CEST October 12, 2022 12:05:57 PM CEST Success
Delete PatchBundles October 12, 2022 12:05:57 PM CEST October 12, 2022 12:05:58 PM CEST Success
At this moment, we can check the status of the components
# odacli describe-component
[2022-10-12 12:06:12 root@oda-deva]# odacli describe-component
System Version
---------------
19.16.0.0.0
System node Name
---------------
oda-deva
Local System Version
---------------
19.16.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.14.0.0.0 19.16.0.0.0
GI
19.14.0.0.220118 19.16.0.0.220719
DB[ OFCCMPRE,T24DEV01,C1IDPDEV,C1IDPPRE,
C1DEVOPS ] 19.14.0.0.220118 19.16.0.0.220719
DCSCONTROLLER
19.16.0.0.0 up-to-date
DCSCLI
19.16.0.0.0 up-to-date
DCSAGENT
19.16.0.0.0 up-to-date
DCSADMIN
19.16.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.0.1.21.a.r138015 5.1.0.20.r145377
BIOS
41080800 41110100
LOCAL CONTROLLER FIRMWARE {
[c3,c4]
20.08.01.18 214.2.271.9
[c5]
80000690 up-to-date
}
SHARED CONTROLLER FIRMWARE
16.00.08.00 up-to-date
FIRMWAREEXPANDER
0310 up-to-date
LOCAL DISK FIRMWARE
N2010121 up-to-date
SHARED DISK FIRMWARE {
[c0d0,c0d1]
N2010121 up-to-date
[c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c1d16,c1d17,c1d18,c1d19,
c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,
c2d8,c2d9,c2d10,c2d11,c2d12,c2d13,c2d14,
c2d15,c2d16,c2d17,c2d18,c2d19] A170 up-to-date
[c1d20,c1d21,c1d22,c1d23,c2d20,c2d21,
c2d22,c2d23] A959 A902
}
HMP
2.4.8.0.600 2.4.8.9.601
System node Name
---------------
oda-devb
Local System Version
---------------
19.16.0.0.0
Component Installed Version Available Version
---------------------------------------- -------------------- --------------------
OAK
19.14.0.0.0 19.16.0.0.0
GI
19.14.0.0.220118 19.16.0.0.220719
DB[ OFCCMPRE,T24DEV01,C1IDPDEV,C1IDPPRE,
C1DEVOPS ] 19.14.0.0.220118 19.16.0.0.220719
DCSCONTROLLER
19.16.0.0.0 up-to-date
DCSCLI
19.16.0.0.0 up-to-date
DCSAGENT
19.16.0.0.0 up-to-date
DCSADMIN
19.16.0.0.0 up-to-date
OS
7.9 up-to-date
ILOM
5.0.1.21.a.r138015 5.1.0.20.r145377
BIOS
41080800 41110100
LOCAL CONTROLLER FIRMWARE {
[c3]
80000690 up-to-date
[c4,c5]
20.08.01.18 214.2.271.9
}
SHARED CONTROLLER FIRMWARE
16.00.08.00 up-to-date
FIRMWAREEXPANDER
0310 up-to-date
LOCAL DISK FIRMWARE
N2010121 up-to-date
SHARED DISK FIRMWARE {
[c0d0,c0d1]
N2010121 up-to-date
[c1d0,c1d1,c1d2,c1d3,c1d4,c1d5,c1d6,
c1d7,c1d8,c1d9,c1d10,c1d11,c1d12,c1d13,
c1d14,c1d15,c1d16,c1d17,c1d18,c1d19,
c2d0,c2d1,c2d2,c2d3,c2d4,c2d5,c2d6,c2d7,
c2d8,c2d9,c2d10,c2d11,c2d12,c2d13,c2d14,
c2d15,c2d16,c2d17,c2d18,c2d19] A170 up-to-date
[c1d20,c1d21,c1d22,c1d23,c2d20,c2d21,
c2d22,c2d23] A959 A902
}
HMP
2.4.8.0.600 2.4.8.9.601
Before updating the server, we do generate the prepatchreport
[2022-10-12 12:07:39 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.16.0.0.0
Job details
----------------------------------------------------------------
ID: a0087a74-994f-4097-b32c-6c024d9a05d7
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
Status: Created
Created: October 12, 2022 12:08:14 PM CEST
Message: Use 'odacli describe-prepatchreport -i a0087a74-994f-4097-b32c-6c024d9a05d7' to check details of results
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
This report should return any error.
[2022-10-12 12:27:25 root@oda-deva]# odacli describe-prepatchreport -i a0087a74-994f-4097-b32c-6c024d9a05d7
Patch pre-check report
------------------------------------------------------------------------
Job ID: a0087a74-994f-4097-b32c-6c024d9a05d7
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
Status: FAILED
Created: October 12, 2022 12:08:14 PM CEST
Result: One or more pre-checks failed for [GI]
Node Name
---------------
oda-deva
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate GI metadata Success Successfully validated GI metadata
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate kernel log level Success Successfully validated the OS log
level
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate Central Inventory Success oraInventory validation passed
Validate patching locks Failed Internal error encountered: Lock on
Central Inventory detected. This may
result in a failure patching the GI.
Validate clones location exist Success Validated clones location
Validate DB start dependencies Success DBs START dependency check passed
Validate DB stop dependencies Success DBs STOP dependency check passed
Evaluate GI patching Success Successfully validated GI patching
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Successfully ran Orachk
Validate command execution Success Validated command execution
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Validated command execution
Node Name
---------------
oda-devb
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate GI metadata Success Successfully validated GI metadata
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate kernel log level Success Successfully validated the OS log
level
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate Central Inventory Success oraInventory validation passed
Validate patching locks Failed Internal error encountered: Lock on
Central Inventory detected. This may
result in a failure patching the GI.
Validate clones location exist Success Validated clones location
Validate DB start dependencies Success DBs START dependency check passed
Validate DB stop dependencies Success DBs STOP dependency check passed
Evaluate GI patching Success Successfully validated GI patching
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Successfully ran Orachk
Validate command execution Success Validated command execution
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Validated command execution
As we can see above we got errors about lock on Central Inventory. We just follow the steps from this Oracle documentation on both nodes
ODA Upgrade PrepatchReport Fails Due To OraInventory Lock Internal Error (Doc ID 2659378.1)
[2022-10-12 13:33:00 root@oda-deva]# cd /u01/app/oraInventory/
[2022-10-12 13:33:15 root@oda-deva]# ls -l
total 24
drwxrwx--- 6 oracle oinstall 4096 Oct 12 12:15 backup
drwxrwx--- 2 oracle oinstall 4096 Oct 12 12:15 ContentsXML
drwxrwx--- 2 oracle oinstall 4096 Oct 12 12:15 locks
drwxrwx--- 7 oracle oinstall 4096 Oct 12 12:14 logs
-rw-rw---- 1 oracle oinstall 56 Apr 20 16:16 oraInst.loc
-rwxrwx--- 1 oracle oinstall 1617 Apr 20 16:16 orainstRoot.sh
[2022-10-12 13:33:18 root@oda-deva]# mv locks locks.221012
And we rerun the prepatch report
[2022-10-12 13:33:40 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.16.0.0.0
Job details
----------------------------------------------------------------
ID: 2d2a2042-31e0-4750-af90-605f6b3bbb20
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
Status: Created
Created: October 12, 2022 1:35:34 PM CEST
Message: Use 'odacli describe-prepatchreport -i 2d2a2042-31e0-4750-af90-605f6b3bbb20' to check details of results
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
And the prepatch report was successful
[2022-10-12 13:45:44 root@oda-deva]# odacli describe-prepatchreport -i 2d2a2042-31e0-4750-af90-605f6b3bbb20
Patch pre-check report
------------------------------------------------------------------------
Job ID: 2d2a2042-31e0-4750-af90-605f6b3bbb20
Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER, SERVER]
Status: SUCCESS
Created: October 12, 2022 1:35:34 PM CEST
Result: All pre-checks succeeded
Node Name
---------------
oda-deva
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate GI metadata Success Successfully validated GI metadata
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate kernel log level Success Successfully validated the OS log
level
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate Central Inventory Success oraInventory validation passed
Validate patching locks Success Validated patching locks
Validate clones location exist Success Validated clones location
Validate DB start dependencies Success DBs START dependency check passed
Validate DB stop dependencies Success DBs STOP dependency check passed
Evaluate GI patching Success Successfully validated GI patching
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Successfully ran Orachk
Validate command execution Success Validated command execution
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Validated command execution
Node Name
---------------
oda-devb
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Verify OS patch Success Verified OS patch
Validate command execution Success Validated command execution
__ILOM__
Validate ILOM server reachable Success Successfully connected with ILOM
server using public IP and USB
interconnect
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is patch location available Success Patch location is available.
Checking Ilom patch Version Success Successfully verified the versions
Patch location validation Success Successfully validated location
Validate command execution Success Validated command execution
__GI__
Validate GI metadata Success Successfully validated GI metadata
Validate supported GI versions Success Validated minimum supported versions.
Validate available space Success Validated free space under /u01
Is clusterware running Success Clusterware is running
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Is system provisioned Success Verified system is provisioned
Validate ASM in online Success ASM is online
Validate kernel log level Success Successfully validated the OS log
level
Validate minimum agent version Success GI patching enabled in current
DCSAGENT version
Validate Central Inventory Success oraInventory validation passed
Validate patching locks Success Validated patching locks
Validate clones location exist Success Validated clones location
Validate DB start dependencies Success DBs START dependency check passed
Validate DB stop dependencies Success DBs STOP dependency check passed
Evaluate GI patching Success Successfully validated GI patching
Validate command execution Success Validated command execution
__ORACHK__
Running orachk Success Successfully ran Orachk
Validate command execution Success Validated command execution
__SERVER__
Validate local patching Success Successfully validated server local
patching
Validate command execution Success Validated command execution
As all checks are fine, we can now run the update server command
[2022-10-12 13:46:19 root@oda-deva]# /opt/oracle/dcs/bin/odacli update-server -v 19.16.0.0.0
{
"jobId" : "a03fce0a-413c-458f-a051-0537109c66f1",
"status" : "Created",
"message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.",
"reports" : [ ],
"createTimestamp" : "October 12, 2022 13:47:11 PM CEST",
"resourceList" : [ ],
"description" : "Server Patching",
"updatedTime" : "October 12, 2022 13:47:11 PM CEST",
"jobType" : null
}
Few times after, the update server was successful
[2022-10-12 15:08:10 root@oda-deva]# odacli describe-job -i a03fce0a-413c-458f-a051-0537109c66f1
Job details
----------------------------------------------------------------
ID: a03fce0a-413c-458f-a051-0537109c66f1
Description: Server Patching
Status: Success
Created: October 12, 2022 1:47:11 PM CEST
Message: Successfully patched GI with RHP
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validating GI user metadata October 12, 2022 1:47:37 PM CEST October 12, 2022 1:47:37 PM CEST Success
Validating GI user metadata October 12, 2022 1:47:37 PM CEST October 12, 2022 1:47:37 PM CEST Success
Validate ILOM server reachable October 12, 2022 1:47:37 PM CEST October 12, 2022 1:47:37 PM CEST Success
Validate ILOM server reachable October 12, 2022 1:47:37 PM CEST October 12, 2022 1:47:37 PM CEST Success
Validate DCS Admin mTLS setup October 12, 2022 1:47:37 PM CEST October 12, 2022 1:47:37 PM CEST Success
Validate DCS Admin mTLS setup October 12, 2022 1:47:37 PM CEST October 12, 2022 1:47:37 PM CEST Success
Configure export clones resource October 12, 2022 1:47:40 PM CEST October 12, 2022 1:47:40 PM CEST Success
Creating repositories using yum October 12, 2022 1:47:43 PM CEST October 12, 2022 1:47:46 PM CEST Success
Updating YumPluginVersionLock rpm October 12, 2022 1:47:46 PM CEST October 12, 2022 1:47:46 PM CEST Success
Applying OS Patches October 12, 2022 1:47:46 PM CEST October 12, 2022 1:59:44 PM CEST Success
Creating repositories using yum October 12, 2022 1:59:45 PM CEST October 12, 2022 1:59:46 PM CEST Success
Applying HMP Patches October 12, 2022 1:59:46 PM CEST October 12, 2022 2:00:05 PM CEST Success
Patch location validation October 12, 2022 2:00:06 PM CEST October 12, 2022 2:00:06 PM CEST Success
Patch location validation October 12, 2022 2:00:06 PM CEST October 12, 2022 2:00:06 PM CEST Success
Oda-hw-mgmt upgrade October 12, 2022 2:00:07 PM CEST October 12, 2022 2:00:40 PM CEST Success
Oda-hw-mgmt upgrade October 12, 2022 2:00:40 PM CEST October 12, 2022 2:01:14 PM CEST Success
OSS Patching October 12, 2022 2:01:14 PM CEST October 12, 2022 2:01:14 PM CEST Success
Applying Firmware Disk Patches October 12, 2022 2:01:18 PM CEST October 12, 2022 2:01:23 PM CEST Success
Applying Firmware Controller Patches October 12, 2022 2:03:17 PM CEST October 12, 2022 2:05:14 PM CEST Success
Checking Ilom patch Version October 12, 2022 2:05:14 PM CEST October 12, 2022 2:05:14 PM CEST Success
Checking Ilom patch Version October 12, 2022 2:05:14 PM CEST October 12, 2022 2:05:15 PM CEST Success
Patch location validation October 12, 2022 2:05:15 PM CEST October 12, 2022 2:05:15 PM CEST Success
Patch location validation October 12, 2022 2:05:15 PM CEST October 12, 2022 2:05:15 PM CEST Success
Save password in Wallet October 12, 2022 2:05:15 PM CEST October 12, 2022 2:05:16 PM CEST Success
Disabling IPMI v2 October 12, 2022 2:05:17 PM CEST October 12, 2022 2:05:19 PM CEST Success
Apply Ilom patch October 12, 2022 2:05:19 PM CEST October 12, 2022 2:17:07 PM CEST Success
Apply Ilom patch October 12, 2022 2:17:07 PM CEST October 12, 2022 2:26:49 PM CEST Success
Copying Flash Bios to Temp location October 12, 2022 2:26:49 PM CEST October 12, 2022 2:26:49 PM CEST Success
Copying Flash Bios to Temp location October 12, 2022 2:26:49 PM CEST October 12, 2022 2:26:49 PM CEST Success
Remove AsrConfBackup File October 12, 2022 2:26:49 PM CEST October 12, 2022 2:26:49 PM CEST Success
Starting the clusterware October 12, 2022 2:28:52 PM CEST October 12, 2022 2:30:39 PM CEST Success
Registering image October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Registering working copy October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Registering image October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Creating GI home directories October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Extract GI clone October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Extract GI clone October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Provisioning Software Only GI with RHP October 12, 2022 2:30:39 PM CEST October 12, 2022 2:30:39 PM CEST Success
Patch GI with RHP October 12, 2022 2:30:39 PM CEST October 12, 2022 2:44:42 PM CEST Success
Updating GIHome version October 12, 2022 2:44:43 PM CEST October 12, 2022 2:44:46 PM CEST Success
Updating GIHome version October 12, 2022 2:44:43 PM CEST October 12, 2022 2:44:47 PM CEST Success
Validate GI availability October 12, 2022 2:44:50 PM CEST October 12, 2022 2:44:51 PM CEST Success
Patch KVM CRS type October 12, 2022 2:44:51 PM CEST October 12, 2022 2:44:51 PM CEST Success
Patch VM vDisks CRS dependencies October 12, 2022 2:44:51 PM CEST October 12, 2022 2:44:51 PM CEST Success
Update System version October 12, 2022 2:44:51 PM CEST October 12, 2022 2:44:51 PM CEST Success
Update System version October 12, 2022 2:44:51 PM CEST October 12, 2022 2:44:51 PM CEST Success
Cleanup JRE Home October 12, 2022 2:44:52 PM CEST October 12, 2022 2:44:52 PM CEST Success
Add SYSNAME in Env October 12, 2022 2:44:52 PM CEST October 12, 2022 2:44:52 PM CEST Success
Starting the clusterware October 12, 2022 2:44:52 PM CEST October 12, 2022 2:44:52 PM CEST Success
Setting ACL for disk groups October 12, 2022 2:44:52 PM CEST October 12, 2022 2:45:00 PM CEST Success
Update previous workarounds October 12, 2022 2:47:21 PM CEST October 12, 2022 2:47:21 PM CEST Success
Update previous workarounds October 12, 2022 2:47:21 PM CEST October 12, 2022 2:47:21 PM CEST Success
PreRebootNode Actions October 12, 2022 2:47:21 PM CEST October 12, 2022 2:48:16 PM CEST Success
PreRebootNode Actions October 12, 2022 2:48:16 PM CEST October 12, 2022 2:49:07 PM CEST Success
Reboot Ilom October 12, 2022 2:49:07 PM CEST October 12, 2022 2:49:07 PM CEST Success
Reboot Ilom October 12, 2022 2:49:07 PM CEST October 12, 2022 2:49:07 PM CEST Success
Now we have to patch the storage
Let’s run the prepatch report for the storage
[2022-10-12 15:08:54 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport -st -v 19.16.0.0.0
Job details
----------------------------------------------------------------
ID: 24fbea1f-8a0c-49bc-b87c-412c64341f62
Description: Patch pre-checks for [STORAGE]
Status: Created
Created: October 12, 2022 3:09:36 PM CEST
Message: Use 'odacli describe-prepatchreport -i 24fbea1f-8a0c-49bc-b87c-412c64341f62' to check details of results
Task Name Start Time End Time Status
Let’s validate the status of the job
[2022-10-12 15:10:22 root@oda-deva]# odacli describe-prepatchreport -i 24fbea1f-8a0c-49bc-b87c-412c64341f62
Patch pre-check report
------------------------------------------------------------------------
Job ID: 24fbea1f-8a0c-49bc-b87c-412c64341f62
Description: Patch pre-checks for [STORAGE]
Status: SUCCESS
Created: October 12, 2022 3:09:36 PM CEST
Result: All pre-checks succeeded
Node Name
---------------
oda-deva
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__STORAGE__
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Patch location validation Success Verified patch location
Patch tag validation Success Verified patch tag
Storage patch tag validation Success Verified storage patch location
Verify ASM disks status Success ASM disks are online
Validate rolling patch Success Rolling mode patching allowed as
there is no expander and controller
upgrade.
Validate command execution Success Validated command execution
Node Name
---------------
oda-devb
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__STORAGE__
Validate patching tag Success Validated patching tag: 19.16.0.0.0.
Patch location validation Success Verified patch location
Patch tag validation Success Verified patch tag
Storage patch tag validation Success Verified storage patch location
Verify ASM disks status Success ASM disks are online
Validate rolling patch Success Rolling mode patching allowed as
there is no expander and controller
upgrade.
Validate command execution Success Validated command execution
Then let’s patch the storage
[2022-10-12 15:10:28 root@oda-deva]# /opt/oracle/dcs/bin/odacli update-storage -v 19.16.0.0.0 --rolling
{
"jobId" : "413bb3b5-77b8-44ae-b1b1-ca3d6ea7fc9e",
"status" : "Created",
"message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart",
"reports" : [ ],
"createTimestamp" : "October 12, 2022 15:11:10 PM CEST",
"resourceList" : [ ],
"description" : "Storage Firmware Patching",
"updatedTime" : "October 12, 2022 15:11:10 PM CEST",
"jobType" : null
}
The operation went fine a few minutes after
[2022-10-12 15:11:20 root@oda-deva]# odacli describe-job -i "413bb3b5-77b8-44ae-b1b1-ca3d6ea7fc9e"
Job details
----------------------------------------------------------------
ID: 413bb3b5-77b8-44ae-b1b1-ca3d6ea7fc9e
Description: Storage Firmware Patching
Status: Success
Created: October 12, 2022 3:11:10 PM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Applying Firmware Disk Patches October 12, 2022 3:11:20 PM CEST October 12, 2022 3:11:26 PM CEST Success
After the server patch, if you decide to update the dbhomes you can do following steps
First list your dbhomes
[2022-10-12 15:11:51 root@oda-deva]# odacli list-dbhomes
ID Name DB Version Home Location Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
a1bfe23e-2569-407b-8b87-7af9f9f586bf OraDB19000_home1 19.14.0.0.220118 /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 CONFIGURED
Run the prepatch report for the dbhomes you want to update
[2022-10-12 15:16:51 root@oda-deva]# /opt/oracle/dcs/bin/odacli create-prepatchreport --dbhome --dbhomeid a1bfe23e-2569-407b-8b87-7af9f9f586bf -v 19.16.0.0.0
Job details
----------------------------------------------------------------
ID: 7bdce971-fc80-4781-8702-458f6f8d2783
Description: Patch pre-checks for [DB, ORACHKDB]: DbHome is OraDB19000_home1
Status: Created
Created: October 12, 2022 3:16:56 PM CEST
Message: Use 'odacli describe-prepatchreport -i 7bdce971-fc80-4781-8702-458f6f8d2783' to check details of results
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Then validate that there is no error in the prepatch report.
And update the dbhome
# /opt/oracle/dcs/bin/odacli update-dbhome --dbhomeid a1bfe23e-2569-407b-8b87-7af9f9f586bf -v 19.16.0.0.0
The job should return SUCCESS
[2022-10-13 15:40:10 root@oda-devb]# odacli describe-job -i 9ff71b6a-cfb1-494a-9fc2-96e903c293e1
Job details
----------------------------------------------------------------
ID: 9ff71b6a-cfb1-494a-9fc2-96e903c293e1
Description: DB Home Patching: Home Id is a1bfe23e-2569-407b-8b87-7af9f9f586bf
Status: Success
Created: October 13, 2022 6:00:35 AM CEST
Message:
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Creating wallet for DB Client October 13, 2022 6:01:45 AM CEST October 13, 2022 6:01:45 AM CEST Success
Patch databases by RHP October 13, 2022 6:01:45 AM CEST October 13, 2022 6:19:09 AM CEST Success
Updating database metadata October 13, 2022 6:19:09 AM CEST October 13, 2022 6:19:09 AM CEST Success
Set log_archive_dest for Database October 13, 2022 6:19:09 AM CEST October 13, 2022 6:19:13 AM CEST Success
Patch databases by RHP October 13, 2022 6:19:13 AM CEST October 13, 2022 6:36:09 AM CEST Success
Updating database metadata October 13, 2022 6:36:09 AM CEST October 13, 2022 6:36:09 AM CEST Success
Set log_archive_dest for Database October 13, 2022 6:36:09 AM CEST October 13, 2022 6:36:13 AM CEST Success
Update System version October 13, 2022 6:36:13 AM CEST October 13, 2022 6:36:13 AM CEST Success
Update System version October 13, 2022 6:36:13 AM CEST October 13, 2022 6:36:13 AM CEST Success
TDE parameter update October 13, 2022 6:36:45 AM CEST October 13, 2022 6:36:45 AM CEST Success
Conclusion
As we can see patching an ODA HA is not different than patching an non-HA ODA. Just some recommendations
-Read the official documentation
-Backup your system before with odabr
-Ensure that you have at least 15G of free space in / /opt and /u01 filesystems