In my previous blog, I described the steps to deploy an ODA 19.8 by reimaging. The ODA was reimaged with the latest available version 19.8 but some components were not up-to-dated.
[[email protected] ~]# odacli describe-component System Version --------------- 19.8.0.0.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 19.8.0.0.0 up-to-date GI 19.8.0.0.200714 up-to-date DCSAGENT 19.8.0.0.0 up-to-date ILOM 4.0.4.38.a.r132148 4.0.4.51.r134837 BIOS 52020500 52021300 OS 7.8 up-to-date FIRMWARECONTROLLER VDV1RL02 VDV1RL04 FIRMWAREDISK 1120 1102 HMP
The patch was downloaded and copied to the ODA and we uncompress the 2 files using unzip
[[email protected] software_oda]# unzip p31481816_198000_Linux-x86-64_1of2.zip Archive: p31481816_198000_Linux-x86-64_1of2.zip extracting: oda-sm-19.8.0.0.0-200718-server1of2.zip y inflating: README.txt [[email protected] software_oda]# ls oda-sm-19.8.0.0.0-200718-server1of2.zip p31481816_198000_Linux-x86-64_2of2.zip p31481816_198000_Linux-x86-64_1of2.zip README.txt [[email protected] software_oda]# unzip p31481816_198000_Linux-x86-64_2of2.zip Archive: p31481816_198000_Linux-x86-64_2of2.zip extracting: oda-sm-19.8.0.0.0-200718-server2of2.zip [[email protected] software_oda]#
We have to update the repository with the 2 archives
[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/software_oda/oda-sm-19.8.0.0.0-200718-server1of2.zip { "jobId" : "1d1a36e0-4630-4f32-a880-7ee2c6d13cf7", "status" : "Created", "message" : "/u01/software_oda/oda-sm-19.8.0.0.0-200718-server1of2.zip", "reports" : [ ], "createTimestamp" : "August 18, 2020 09:40:01 AM CEST", "resourceList" : [ ], "description" : "Repository Update", "updatedTime" : "August 18, 2020 09:40:01 AM CEST" } [[email protected] ~]#
The status of the job should return SUCCESS
[[email protected] ~]# /opt/oracle/dcs/bin/odacli describe-job -i "1d1a36e0-4630-4f32-a880-7ee2c6d13cf7" Job details ---------------------------------------------------------------- ID: 1d1a36e0-4630-4f32-a880-7ee2c6d13cf7 Description: Repository Update Status: Success Created: August 18, 2020 9:40:01 AM CEST Message: /u01/software_oda/oda-sm-19.8.0.0.0-200718-server1of2.zip Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- [[email protected] ~]#
We do the same thing with the second archive
[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-repository -f /u01/software_oda/oda-sm-19.8.0.0.0-200718-server2of2.zip { "jobId" : "60bd7144-d8a2-4f39-b58f-ae73dc0aecb6", "status" : "Created", "message" : "/u01/software_oda/oda-sm-19.8.0.0.0-200718-server2of2.zip", "reports" : [ ], "createTimestamp" : "August 18, 2020 09:41:34 AM CEST", "resourceList" : [ ], "description" : "Repository Update", "updatedTime" : "August 18, 2020 09:41:34 AM CEST" } [[email protected] ~]#
And we verify the satus is returning SUCCESS
[[email protected] ~]# /opt/oracle/dcs/bin/odacli describe-job -i "60bd7144-d8a2-4f39-b58f-ae73dc0aecb6"
After updating the repository we update the DCS agent
[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-dcsagent -v 19.8.0.0.0 { "jobId" : "2f7dbdda-3fc1-4dbf-8fe9-d14e1514679e", "status" : "Created", "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands", "reports" : [ ], "createTimestamp" : "August 18, 2020 09:43:31 AM CEST", "resourceList" : [ ], "description" : "DcsAgent patching", "updatedTime" : "August 18, 2020 09:43:31 AM CEST" } [[email protected] ~]#
The status should return SUCCESS
[[email protected] ~]# /opt/oracle/dcs/bin/odacli describe-job -i "2f7dbdda-3fc1-4dbf-8fe9-d14e1514679e" Job details ---------------------------------------------------------------- ID: 2f7dbdda-3fc1-4dbf-8fe9-d14e1514679e Description: DcsAgent patching Status: Success Created: August 18, 2020 9:43:31 AM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- dcs-agent upgrade to version 19.8.0.0.0 August 18, 2020 9:43:31 AM CEST August 18, 2020 9:43:31 AM CEST Success Update System version August 18, 2020 9:43:31 AM CEST August 18, 2020 9:43:31 AM CEST Success [[email protected] ~]#
The DCS components should be also updated
[[email protected] ~]# /opt/oracle/dcs/bin/odacli update-dcscomponents -v 19.8.0.0.0 { "jobId" : "f4359690-a910-484f-9382-b3b94b590c6b", "status" : "Success", "message" : null, "reports" : null, "createTimestamp" : "August 18, 2020 09:47:03 AM CEST", "description" : "Job completed and is not part of Agent job list", "updatedTime" : "August 18, 2020 09:47:03 AM CEST" } [[email protected] ~]#
Just not that a describe-job for the id above will return error. You just must have SUCCESS in the status
Now we can generate a prepatch-report to correct the eventual errors while patching
[[email protected] ~]# /opt/oracle/dcs/bin/odacli create-prepatchreport -s -v 19.8.0.0.0 Job details ---------------------------------------------------------------- ID: 06fdc0c4-7a1e-45aa-b930-dce35ab978a9 Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER] Status: Created Created: August 18, 2020 9:49:39 AM CEST Message: Use 'odacli describe-prepatchreport -i 06fdc0c4-7a1e-45aa-b930-dce35ab978a9' to check details of results Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- [[email protected] ~]#
As indicated in the output above, we run the describe-prepatchreport to have the results
[[email protected] ~]# odacli describe-prepatchreport -i 06fdc0c4-7a1e-45aa-b930-dce35ab978a9 Patch pre-check report ------------------------------------------------------------------------ Job ID: 06fdc0c4-7a1e-45aa-b930-dce35ab978a9 Description: Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER] Status: FAILED Created: August 18, 2020 9:49:39 AM CEST Result: One or more pre-checks failed for [ORACHK] Node Name --------------- serveroda Pre-Check Status Comments ------------------------------ -------- -------------------------------------- __OS__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Verify OS patch Success There are no packages available for an update Validate command execution Success Validated command execution __ILOM__ Validate supported versions Success Validated minimum supported versions. Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is patch location available Success Patch location is available. Checking Ilom patch Version Success Successfully verified the versions Patch location validation Success Successfully validated location Validate command execution Success Validated command execution __GI__ Validate supported GI versions Success Validated minimum supported versions. Validate available space Success Validated free space under /u01 Is clusterware running Success Clusterware is running Validate patching tag Success Validated patching tag: 19.8.0.0.0. Is system provisioned Success Verified system is provisioned Validate ASM in online Success ASM is online Validate minimum agent version Success GI patching enabled in current DCSAGENT version Validate GI patch metadata Success Validated patching tag: 19.8.0.0.0. Validate clones location exist Success Validated clones location Is patch location available Success Patch location is available. Patch location validation Success Successfully validated location Patch verification Success Patch 31281355 already applied on gi home /u01/app/19.0.0.0/grid on node serveroda Validate Opatch update Success Not updating opatch as patch already applied on node serveroda Patch conflict check Success Not analyzing patch as patch already applied on node serveroda Validate command execution Success Validated command execution __ORACHK__ Running orachk Failed Orachk validation failed: . Validate command execution Success Validated command execution CSS disktimeout Failed CSS disktimeout is not set to the default value of 200 Verify Cluster Failed Cluster Synchronization Services Synchronization Services (CSS) misscount not set to (CSS) misscount value recommended value Software home Failed Software home check failed
As we can see there are some errors because the system is expecting default values for some parameters.
For example the value of the disktimeout is set to 251 instead of the expected value by the prepatch check 200.
[[email protected] log]# crsctl get css disktimeout CRS-4678: Successful get disktimeout 251 for Cluster Synchronization Services.
So I may think that these errors can be ignored. And then I decide to continue by updating the server with the patch
[[email protected] log]# /opt/oracle/dcs/bin/odacli update-server -v 19.8.0.0.0 { "jobId" : "de8ce20e-81b1-45a5-bd21-7271dc4a8d7f", "status" : "Created", "message" : "Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.", "reports" : [ ], "createTimestamp" : "August 18, 2020 10:19:46 AM CEST", "resourceList" : [ ], "description" : "Server Patching", "updatedTime" : "August 18, 2020 10:19:46 AM CEST" } [[email protected] log]#
A few minutes after the patch was successfully applied
[[email protected] log]# odacli describe-job -i "de8ce20e-81b1-45a5-bd21-7271dc4a8d7f" Job details ---------------------------------------------------------------- ID: de8ce20e-81b1-45a5-bd21-7271dc4a8d7f Description: Server Patching Status: Success Created: August 18, 2020 10:19:46 AM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Patch location validation August 18, 2020 10:19:53 AM CEST August 18, 2020 10:19:54 AM CEST Success dcs-controller upgrade August 18, 2020 10:19:54 AM CEST August 18, 2020 10:19:54 AM CEST Success Patch location validation August 18, 2020 10:19:54 AM CEST August 18, 2020 10:19:54 AM CEST Success dcs-cli upgrade August 18, 2020 10:19:54 AM CEST August 18, 2020 10:19:54 AM CEST Success Creating repositories using yum August 18, 2020 10:19:54 AM CEST August 18, 2020 10:19:56 AM CEST Success Updating YumPluginVersionLock rpm August 18, 2020 10:19:56 AM CEST August 18, 2020 10:19:56 AM CEST Success Applying OS Patches August 18, 2020 10:19:56 AM CEST August 18, 2020 10:19:57 AM CEST Success Creating repositories using yum August 18, 2020 10:19:57 AM CEST August 18, 2020 10:19:57 AM CEST Success Applying HMP Patches August 18, 2020 10:19:57 AM CEST August 18, 2020 10:19:58 AM CEST Success Patch location validation August 18, 2020 10:19:58 AM CEST August 18, 2020 10:19:58 AM CEST Success oda-hw-mgmt upgrade August 18, 2020 10:19:58 AM CEST August 18, 2020 10:19:58 AM CEST Success OSS Patching August 18, 2020 10:19:58 AM CEST August 18, 2020 10:19:58 AM CEST Success Applying Firmware Disk Patches August 18, 2020 10:19:58 AM CEST August 18, 2020 10:20:05 AM CEST Success Applying Firmware Expander Patches August 18, 2020 10:20:05 AM CEST August 18, 2020 10:20:10 AM CEST Success Applying Firmware Controller Patches August 18, 2020 10:20:10 AM CEST August 18, 2020 10:20:16 AM CEST Success Checking Ilom patch Version August 18, 2020 10:20:17 AM CEST August 18, 2020 10:20:19 AM CEST Success Patch location validation August 18, 2020 10:20:19 AM CEST August 18, 2020 10:20:20 AM CEST Success Save password in Wallet August 18, 2020 10:20:21 AM CEST August 18, 2020 10:20:21 AM CEST Success Apply Ilom patch August 18, 2020 10:20:21 AM CEST August 18, 2020 10:22:16 AM CEST Success Copying Flash Bios to Temp location August 18, 2020 10:22:16 AM CEST August 18, 2020 10:22:16 AM CEST Success Starting the clusterware August 18, 2020 10:22:16 AM CEST August 18, 2020 10:24:38 AM CEST Success clusterware patch verification August 18, 2020 10:24:39 AM CEST August 18, 2020 10:24:49 AM CEST Success Patch location validation August 18, 2020 10:24:49 AM CEST August 18, 2020 10:24:49 AM CEST Success Opatch update August 18, 2020 10:24:49 AM CEST August 18, 2020 10:24:49 AM CEST Success Patch conflict check August 18, 2020 10:24:49 AM CEST August 18, 2020 10:24:49 AM CEST Success clusterware upgrade August 18, 2020 10:24:49 AM CEST August 18, 2020 10:24:49 AM CEST Success Updating GiHome version August 18, 2020 10:24:49 AM CEST August 18, 2020 10:24:52 AM CEST Success Update System version August 18, 2020 10:24:52 AM CEST August 18, 2020 10:24:52 AM CEST Success preRebootNode Actions August 18, 2020 10:24:52 AM CEST August 18, 2020 10:24:52 AM CEST Success Reboot Ilom August 18, 2020 10:24:52 AM CEST August 18, 2020 10:24:52 AM CEST Success [[email protected] log]#
Then we update the storage
[[email protected] log]# /opt/oracle/dcs/bin/odacli update-storage -v 19.8.0.0.0 { "jobId" : "c6e86978-8869-4b24-8d3f-44a8c988c5a4", "status" : "Created", "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart", "reports" : [ ], "createTimestamp" : "August 18, 2020 10:27:31 AM CEST", "resourceList" : [ ], "description" : "Storage Firmware Patching", "updatedTime" : "August 18, 2020 10:27:31 AM CEST" } [[email protected] log]#
The status was SUCCESS and the server is restarted
[[email protected] log]# odacli describe-job -i "c6e86978-8869-4b24-8d3f-44a8c988c5a4 Job details ---------------------------------------------------------------- ID: c6e86978-8869-4b24-8d3f-44a8c988c5a4 Description: Storage Firmware Patching Status: Success Created: August 18, 2020 10:27:31 AM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Applying Firmware Disk Patches August 18, 2020 10:27:36 AM CEST August 18, 2020 10:27:40 AM CEST Success Applying Firmware Controller Patches August 18, 2020 10:27:40 AM CEST August 18, 2020 10:31:38 AM CEST Success preRebootNode Actions August 18, 2020 10:31:38 AM CEST August 18, 2020 10:31:38 AM CEST Success Reboot Ilom August 18, 2020 10:31:38 AM CEST August 18, 2020 10:31:38 AM CEST Success [[email protected] log]# [ OK ] Started Show Plymouth Power Off Screen. [ OK ] Stopped (null). [ OK ] Stopped Oracle High Availability Services. [ OK ] Stopped Availability of block devices. [ OK ] Started Restore /run/initramfs. [ OK ] Stopped Dynamic System Tuning Daemon.
After the reboot I connect and I remark that all components were updated except the BIOS and the ILOM. Then I opened a SR with Oracle and they asked me to follow steps in the following documnent
ODA (Oracle Database Appliance): OAK Bundle Patch failing on ILOM/BIOS component apply (Doc ID 1427885.1), what I did
First I stop the CRS stack
[[email protected] grid]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'serveroda' CRS-2673: Attempting to stop 'ora.crsd' on 'serveroda' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'serveroda' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'serveroda' CRS-2673: Attempting to stop 'ora.cvu' on 'serveroda' CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'serveroda' CRS-2673: Attempting to stop 'ora.RECO.dg' on 'serveroda' CRS-2673: Attempting to stop 'ora.data.commonstore.acfs' on 'serveroda' CRS-2673: Attempting to stop 'ora.qosmserver' on 'serveroda' CRS-2673: Attempting to stop 'ora.chad' on 'serveroda' CRS-2677: Stop of 'ora.RECO.dg' on 'serveroda' succeeded CRS-2677: Stop of 'ora.cvu' on 'serveroda' succeeded CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.serveroda.vip' on 'serveroda' CRS-2677: Stop of 'ora.serveroda.vip' on 'serveroda' succeeded CRS-2677: Stop of 'ora.data.commonstore.acfs' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.DATA.COMMONSTORE.advm' on 'serveroda' CRS-2677: Stop of 'ora.DATA.COMMONSTORE.advm' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'serveroda' CRS-2673: Attempting to stop 'ora.proxy_advm' on 'serveroda' CRS-2677: Stop of 'ora.DATA.dg' on 'serveroda' succeeded CRS-2677: Stop of 'ora.chad' on 'serveroda' succeeded CRS-2677: Stop of 'ora.qosmserver' on 'serveroda' succeeded CRS-2677: Stop of 'ora.proxy_advm' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'serveroda' CRS-2677: Stop of 'ora.asm' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'serveroda' CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'serveroda' CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'serveroda' succeeded CRS-33677: Stop of resource group 'ora.asmgroup' on server 'serveroda' succeeded. CRS-2673: Attempting to stop 'ora.ons' on 'serveroda' CRS-2677: Stop of 'ora.ons' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'serveroda' CRS-2677: Stop of 'ora.net1.network' on 'serveroda' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'serveroda' has completed CRS-2677: Stop of 'ora.crsd' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'serveroda' CRS-2673: Attempting to stop 'ora.crf' on 'serveroda' CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'serveroda' CRS-2673: Attempting to stop 'ora.mdnsd' on 'serveroda' CRS-2677: Stop of 'ora.crf' on 'serveroda' succeeded CRS-2677: Stop of 'ora.drivers.acfs' on 'serveroda' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'serveroda' succeeded CRS-2677: Stop of 'ora.asm' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'serveroda' CRS-2673: Attempting to stop 'ora.evmd' on 'serveroda' CRS-2677: Stop of 'ora.ctssd' on 'serveroda' succeeded CRS-2677: Stop of 'ora.evmd' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'serveroda' CRS-2677: Stop of 'ora.cssd' on 'serveroda' succeeded CRS-2673: Attempting to stop 'ora.driver.afd' on 'serveroda' CRS-2673: Attempting to stop 'ora.gipcd' on 'serveroda' CRS-2673: Attempting to stop 'ora.gpnpd' on 'serveroda' CRS-2677: Stop of 'ora.driver.afd' on 'serveroda' succeeded CRS-2677: Stop of 'ora.gipcd' on 'serveroda' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'serveroda' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'serveroda' has completed CRS-4133: Oracle High Availability Services has been stopped. [[email protected] grid]#
And then we run the command ipmiflash
[[email protected] ~]# cd /opt/oracle/oak/pkgrepos/ilom/x8-2l/4.0.4.51.r134837/ [[email protected] 4.0.4.51.r134837]# ls componentmetadata.xml ILOM-4_0_4_51_r134837-ORACLE_SERVER_X8-2L.pkg [[email protected] 4.0.4.51.r134837]# [[email protected] 4.0.4.51.r134837]# ipmiflash -v write ILOM-4_0_4_51_r134837-ORACLE_SERVER_X8-2L.pkg force script config delaybios warning=0
A few moment after the upgrade was done (around 30 minutes)
Uploading Firmware over Linux OpenIPMI Interface... 64 byte packets More robust algorithm supported. 31968K total 31968K [sending...] data chunk #595201, size 05779K [sending...] data chunk #70366, size 55 Waiting for transfer to finish.......... Sending upgrade command Waiting for upgrade to start.......... Waiting for upgrade to complete........................................................................... Upgrade completed You have new mail in /var/spool/mail/root [[email protected] 4.0.4.51.r134837]# reconnecting to console...
After the reconnection we then verify the version
[[email protected] 4.0.4.51.r134837]# cd [[email protected] ~]# /usr/bin/ipmitool sunoem version Version: 4.0.4.51 r134837 [[email protected] ~]# /usr/bin/ipmitool sunoem getval /SYS/MB/BIOS/fru_version Target Value: 52020500 [[email protected] ~]#
As we can see, now the ILOM is updated and the target value for the BIOS is 52020500 which is the expected value. Then we stop the server
[[email protected] ~]# init 0
4 or 5 minutes after we start the node via the ILOM and then validate that everything is fine
Host Management >>>>> Power management >>>>>Power on and save
[[email protected] ~]# odacli describe-component System Version --------------- 19.8.0.0.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 19.8.0.0.0 up-to-date GI 19.8.0.0.200714 up-to-date DCSAGENT 19.8.0.0.0 up-to-date ILOM 4.0.4.51.r134837 up-to-date BIOS 52021300 up-to-date OS 7.8 up-to-date FIRMWARECONTROLLER VDV1RL04 up-to-date FIRMWAREDISK 1120 1102 HMP 2.4.5.0.1 up-to-date [[email protected] ~]#
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/04/Slide56.png)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/05/stephane-savorgnano.png)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/04/Slide59.png)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/04/Slide37.png)