{"id":15062,"date":"2020-11-05T09:35:27","date_gmt":"2020-11-05T08:35:27","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/"},"modified":"2020-11-05T09:35:27","modified_gmt":"2020-11-05T08:35:27","slug":"patching-oracle-database-appliance-to-19-9","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/","title":{"rendered":"Patching Oracle Database Appliance to 19.9"},"content":{"rendered":"<p>ODA 19.9 has just been released for Bare Metal yesterday, and I had the opportunity to already patch a customer production ODA to this latest version. Through this blog I wanted to share my experience on patching an ODA to 19.9 as well as a new tricky skip-orachk option.<\/p>\n<p><!--more--><\/p>\n<h3>Patching requirement<\/h3>\n<p>To patch the Bare Metal ODA to 19.9 version (patch 31922078), we need to be in either 19.5, 19.6, 19.7 or 19.8 version. This is described in the <a href=\"https:\/\/docs.oracle.com\/en\/engineered-systems\/oracle-database-appliance\/19.9\/cmtrn\/oda-patches.html#GUID-220DA05B-0F52-4EDA-84C9-BFD15F43802D\">ODA documentation<\/a>.<\/p>\n<p>First of all we need to ensure we have enough space on \/, \/u01 and \/opt file systems. At least 20 GB should be available. If not, we can do some cleaning or extend the LVM partitions.<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">\n[root@ODA01 \/]# df -h \/ \/u01 \/opt\nFilesystem                          Size  Used Avail Use% Mounted on\n\/dev\/mapper\/VolGroupSys-LogVolRoot   30G  9.5G   19G  34% \/\n\/dev\/mapper\/VolGroupSys-LogVolU01    99G   55G   40G  59% \/u01\n\/dev\/mapper\/VolGroupSys-LogVolOpt    75G   43G   29G  60% \/opt\n<\/pre>\n<p><\/br><br \/>\nThen we will check that no hardware failure is existing on the ODA. This can be checked with the ILOM GUI or using a ssh connection on the ILOM :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,13,16]\">\n-&gt; show \/SP\/faultmgmt\n\n \/SP\/faultmgmt\n    Targets:\n        shell\n\n    Properties:\n\n    Commands:\n        cd\n        show\n\n-&gt; start \/SP\/faultmgmt\/shell\nAre you sure you want to start \/SP\/faultmgmt\/shell (y\/n)? y\n\nfaultmgmtsp&gt; fmadm faulty\nNo faults found\n<\/pre>\n<p><\/br><br \/>\nRecommendation is to use odabr tool and perform a snapshot backup :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,36]\">\n[root@ODA01 \/]# \/opt\/odabr\/odabr backup -snap\nINFO: 2020-11-04 16:30:42: Please check the logfile '\/opt\/odabr\/out\/log\/odabr_37159.log' for more details\n\n\n--------------------------------------------------------\nodabr - ODA node Backup Restore\nAuthor: Ruggero Citton \nRAC Pack, Cloud Innovation and Solution Engineering Team\nCopyright Oracle, Inc. 2013, 2019\nVersion: 2.0.1-47\n--------------------------------------------------------\n\nINFO: 2020-11-04 16:30:42: Checking superuser\nINFO: 2020-11-04 16:30:42: Checking Bare Metal\nINFO: 2020-11-04 16:30:42: Removing existing LVM snapshots\nWARNING: 2020-11-04 16:30:42: LVM snapshot for 'opt' does not exist\nWARNING: 2020-11-04 16:30:42: LVM snapshot for 'u01' does not exist\nWARNING: 2020-11-04 16:30:42: LVM snapshot for 'root' does not exist\nINFO: 2020-11-04 16:30:42: Checking LVM size\nINFO: 2020-11-04 16:30:42: Doing a snapshot backup only\nINFO: 2020-11-04 16:30:42: Boot device backup\nINFO: 2020-11-04 16:30:42: ...getting boot device\nINFO: 2020-11-04 16:30:42: ...making boot device backup\nINFO: 2020-11-04 16:30:44: ...boot device backup saved as '\/opt\/odabr\/out\/hbi\/boot.img'\nINFO: 2020-11-04 16:30:44: Getting EFI device\nINFO: 2020-11-04 16:30:44: ...making efi device backup\nINFO: 2020-11-04 16:30:46: EFI device backup saved as '\/opt\/odabr\/out\/hbi\/efi.img'\nINFO: 2020-11-04 16:30:46: OCR backup\nINFO: 2020-11-04 16:30:47: ...ocr backup saved as '\/opt\/odabr\/out\/hbi\/ocrbackup_37159.bck'\nINFO: 2020-11-04 16:30:47: Making LVM snapshot backup\nSUCCESS: 2020-11-04 16:30:49: ...snapshot backup for 'opt' created successfully\nSUCCESS: 2020-11-04 16:30:49: ...snapshot backup for 'u01' created successfully\nSUCCESS: 2020-11-04 16:30:49: ...snapshot backup for 'root' created successfully\nSUCCESS: 2020-11-04 16:30:49: LVM snapshots backup done successfully\n\n[root@ODA01 \/]# \/opt\/odabr\/odabr infosnap\n\n--------------------------------------------------------\nodabr - ODA node Backup Restore\nAuthor: Ruggero Citton \nRAC Pack, Cloud Innovation and Solution Engineering Team\nCopyright Oracle, Inc. 2013, 2019\nVersion: 2.0.1-47\n--------------------------------------------------------\n\n\nLVM snap name         Status                COW Size              Data%\n-------------         ----------            ----------            ------\nroot_snap             active                30.00 GiB             0.01%\nopt_snap              active                60.00 GiB             0.01%\nu01_snap              active                100.00 GiB            0.01%\n<\/pre>\n<p><\/br><br \/>\nWe can as well run orachk excluding the rdbms checks :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,3]\">\n[root@ODA01 \/]# cd \/opt\/oracle\/dcs\/oracle.ahf\/orachk\n\n[root@ODA01 orachk]# .\/orachk -nordbms\n\n.  .  .  .  .  .\n\nEither Cluster Verification Utility pack (cvupack) does not exist at \/opt\/oracle\/dcs\/oracle.ahf\/common\/cvu or it is an old or invalid cvupack\n\nChecking Cluster Verification Utility (CVU) version at CRS Home - \/u01\/app\/19.0.0.0\/grid\n\nThis version of Cluster Verification Utility (CVU) was released on 10-Mar-2020 and it is older than 180 days. It is highly recommended that you download the latest version of CVU from MOS patch 30166242 to ensure the highest level of accuracy of the data contained within the report\n\nDo you want to download latest version of Cluster Verification Utility (CVU) from my oracle support? [y\/n] [y] n\n\nRunning older version of Cluster Verification Utility (CVU) from CRS Home - \/u01\/app\/19.0.0.0\/grid\n\nChecking Status of Oracle Software Stack - Clusterware, ASM, RDBMS on oda01\n\n.\n.  .  . . . .  .  .  .\n-------------------------------------------------------------------------------------------------------\n                                                 Oracle Stack Status\n-------------------------------------------------------------------------------------------------------\n  Host Name       CRS Installed  RDBMS Installed    CRS UP    ASM UP  RDBMS UP    DB Instance Name\n-------------------------------------------------------------------------------------------------------\noda01               Yes           No          Yes      Yes       No\n-------------------------------------------------------------------------------------------------------\n.\n.  .  .  .  .  .\n\n\n.\n.\n.\n.\n\n.\n\n\n\n*** Checking Best Practice Recommendations ( Pass \/ Warning \/ Fail ) ***\n\n\nCollections and audit checks log file is\n\/opt\/oracle\/dcs\/oracle.ahf\/data\/oda01\/orachk\/orachk_oda01_110420_163217\/log\/orachk.log\n\n============================================================\n           Node name - oda01\n============================================================\n\n Collecting - ASM Disk Group for Infrastructure Software and Configuration\n Collecting - ASM Diskgroup Attributes\n Collecting - ASM initialization parameters\n Collecting - Disk I\/O Scheduler on Linux\n Collecting - Interconnect network card speed\n Collecting - Kernel parameters\n Collecting - Maximum number of semaphore sets on system\n Collecting - Maximum number of semaphores on system\n Collecting - Maximum number of semaphores per semaphore set\n Collecting - OS Packages\n Collecting - Patches for Grid Infrastructure\n Collecting - number of semaphore operations per semop system call\n Collecting - CRS user limits configuration\n Collecting - Database Server Infrastructure Software and Configuration\n Collecting - umask setting for GI owner\n\nData collections completed. Checking best practices on oda01.\n------------------------------------------------------------\n\n INFO =&gt;     Oracle Database Appliance Best Practice References\n INFO =&gt;     Oracle Data Pump Best practices.\n INFO =&gt;     Important Storage Minimum Requirements for Grid &amp; Database Homes\n WARNING =&gt;  soft or hard memlock are not configured according to recommendation\n INFO =&gt;     CSS disktimeout is not set to the default value\n WARNING =&gt;  OCR is not being backed up daily\n INFO =&gt;     CSS misscount is not set to the default value of 30\n INFO =&gt;     Jumbo frames (MTU &gt;= 9000) are not configured for interconnect\n INFO =&gt;     Information about hanganalyze and systemstate dump\n WARNING =&gt;  One or more diskgroups from v$asm_diskgroups are not registered in clusterware registry\nBest Practice checking completed. Checking recommended patches on oda01\n--------------------------------------------------------------------------------\nCollecting patch inventory on CRS_HOME \/u01\/app\/19.0.0.0\/grid\nCollecting patch inventory on ASM_HOME \/u01\/app\/19.0.0.0\/grid\n\n------------------------------------------------------------\n                      CLUSTERWIDE CHECKS\n------------------------------------------------------------\n\n------------------------------------------------------------\nDetailed report (html) -  \/opt\/oracle\/dcs\/oracle.ahf\/data\/oda01\/orachk\/orachk_oda01_110420_163217\/orachk_oda01_110420_163217.html\n\nUPLOAD [if required] - \/opt\/oracle\/dcs\/oracle.ahf\/data\/oda01\/orachk\/orachk_oda01_110420_163217.zip\n<\/pre>\n<p>Then we need to ensure to have a good backup for the opened databases that will run on the ODA. If we are patching an ODA having High Availability (Data Guard for EE edition or dbvisit for SE edition), we will ensure to have run switchover and have only standby databases running on the ODA. And we need to stop the databases&#8217; synchronization in that case.<\/p>\n<h3>Patching the ODA to 19.9<\/h3>\n<p>Once this requirements are met, we can start the patching.<\/p>\n<p>We first need to unzip the downloaded patch files. The patch 31922078 files will be downloaded from <a href=\"https:\/\/support.oracle.com\/portal\/\">oracle web support portal<\/a>.<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,3,8,13,17,25,27]\">\n[root@ODA01 orachk]# cd \/u01\/app\/patch\/\n\n[root@ODA01 patch]# ls -ltrh\ntotal 16G\n-rw-r--r-- 1 root root 6.7G Nov  4 14:11 p31922078_199000_Linux-x86-64_2of2.zip\n-rw-r--r-- 1 root root 9.2G Nov  4 15:17 p31922078_199000_Linux-x86-64_1of2.zip\n\n[root@ODA01 patch]# unzip p31922078_199000_Linux-x86-64_1of2.zip\nArchive:  p31922078_199000_Linux-x86-64_1of2.zip\n extracting: oda-sm-19.9.0.0.0-201023-server1of2.zip\n  inflating: README.txt\n  \n[root@ODA01 patch]# unzip p31922078_199000_Linux-x86-64_2of2.zip\nArchive:  p31922078_199000_Linux-x86-64_2of2.zip\n extracting: oda-sm-19.9.0.0.0-201023-server2of2.zip\n\n[root@ODA01 patch]# ls -ltrh\ntotal 32G\n-rw-r--r-- 1 root root 9.2G Oct 29 04:51 oda-sm-19.9.0.0.0-201023-server1of2.zip\n-rw-r--r-- 1 root root 6.7G Oct 29 04:53 oda-sm-19.9.0.0.0-201023-server2of2.zip\n-rw-r--r-- 1 root root  190 Oct 29 06:17 README.txt\n-rw-r--r-- 1 root root 6.7G Nov  4 14:11 p31922078_199000_Linux-x86-64_2of2.zip\n-rw-r--r-- 1 root root 9.2G Nov  4 15:17 p31922078_199000_Linux-x86-64_1of2.zip\n\n[root@ODA01 patch]# rm -f p31922078_199000_Linux-x86-64_2of2.zip\n\n[root@ODA01 patch]# rm -f p31922078_199000_Linux-x86-64_1of2.zip\n<\/pre>\n<p><\/br><br \/>\nWe can then update the ODA repository with the patch files :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,13,26,38,51]\">\n[root@ODA01 patch]# odacli update-repository -f \/u01\/app\/patch\/oda-sm-19.9.0.0.0-201023-server1of2.zip\n{\n  \"jobId\" : \"0c23cb4e-2455-4ad2-832b-168edce2f40c\",\n  \"status\" : \"Created\",\n  \"message\" : \"\/u01\/app\/patch\/oda-sm-19.9.0.0.0-201023-server1of2.zip\",\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 16:55:52 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Repository Update\",\n  \"updatedTime\" : \"November 04, 2020 16:55:52 PM CET\"\n}\n\n[root@ODA01 patch]# odacli describe-job -i \"0c23cb4e-2455-4ad2-832b-168edce2f40c\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  0c23cb4e-2455-4ad2-832b-168edce2f40c\n            Description:  Repository Update\n                 Status:  Success\n                Created:  November 4, 2020 4:55:52 PM CET\n                Message:  \/u01\/app\/patch\/oda-sm-19.9.0.0.0-201023-server1of2.zip\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@ODA01 patch]# odacli update-repository -f \/u01\/app\/patch\/oda-sm-19.9.0.0.0-201023-server2of2.zip\n{\n  \"jobId\" : \"04ecd45d-6b92-475c-acd9-202f0137474f\",\n  \"status\" : \"Created\",\n  \"message\" : \"\/u01\/app\/patch\/oda-sm-19.9.0.0.0-201023-server2of2.zip\",\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 16:58:05 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Repository Update\",\n  \"updatedTime\" : \"November 04, 2020 16:58:05 PM CET\"\n}\n\n[root@ODA01 patch]# odacli describe-job -i \"04ecd45d-6b92-475c-acd9-202f0137474f\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  04ecd45d-6b92-475c-acd9-202f0137474f\n            Description:  Repository Update\n                 Status:  Success\n                Created:  November 4, 2020 4:58:05 PM CET\n                Message:  \/u01\/app\/patch\/oda-sm-19.9.0.0.0-201023-server2of2.zip\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n\n[root@ODA01 patch]# odacli list-jobs | head -n 3;  odacli list-jobs | tail -n 3\n\nID                                       Description                                                                 Created                             Status\n---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------\n0c23cb4e-2455-4ad2-832b-168edce2f40c     Repository Update                                                           November 4, 2020 4:55:52 PM CET     Success\n04ecd45d-6b92-475c-acd9-202f0137474f     Repository Update                                                           November 4, 2020 4:58:05 PM CET     Success\n<\/pre>\n<p><\/br><br \/>\nWe can already clean up the patch folder as the files are not needed any more :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,7,9]\">\n[root@ODA01 patch]# ls -ltrh\ntotal 16G\n-rw-r--r-- 1 root root 9.2G Oct 29 04:51 oda-sm-19.9.0.0.0-201023-server1of2.zip\n-rw-r--r-- 1 root root 6.7G Oct 29 04:53 oda-sm-19.9.0.0.0-201023-server2of2.zip\n-rw-r--r-- 1 root root  190 Oct 29 06:17 README.txt\n\n[root@ODA01 patch]# rm -f *.zip\n\n[root@ODA01 patch]# rm -f README.txt\n<\/pre>\n<p><\/br><br \/>\nWe will check the current version and available new version :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">\n[root@ODA01 patch]# odacli describe-component\nSystem Version\n---------------\n19.6.0.0.0\n\nComponent                                Installed Version    Available Version\n---------------------------------------- -------------------- --------------------\nOAK                                       19.6.0.0.0            19.9.0.0.0\nGI                                        19.6.0.0.200114       19.9.0.0.201020\nDB                                        18.7.0.0.190716       18.12.0.0.201020\nDCSAGENT                                  19.6.0.0.0            19.9.0.0.0\nILOM                                      4.0.4.51.r133528      5.0.1.21.r136383\nBIOS                                      52021000              52030400\nOS                                        7.7                   7.8\nFIRMWARECONTROLLER                        VDV1RL02              VDV1RL04\nFIRMWAREDISK                              1102                  1132\nHMP                                       2.4.5.0.1             2.4.7.0.1\n<\/pre>\n<p><\/br><br \/>\nI&#8217;m usually stopping the databases at that time. It is not mandatory, but I personally prefer. This can be achieved by stopping each database with srvctl stop database command or srvctl stop home command to stop all databases from same rdbms home.<\/p>\n<p>Now we can update the dcs-agent :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,13]\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli update-dcsagent -v 19.9.0.0.0\n{\n  \"jobId\" : \"fa6c5e53-b0b7-470e-b856-ccf19a0305ef\",\n  \"status\" : \"Created\",\n  \"message\" : \"Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands\",\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 17:02:53 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"DcsAgent patching\",\n  \"updatedTime\" : \"November 04, 2020 17:02:53 PM CET\"\n}\n\n[root@ODA01 patch]# odacli describe-job -i \"fa6c5e53-b0b7-470e-b856-ccf19a0305ef\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  fa6c5e53-b0b7-470e-b856-ccf19a0305ef\n            Description:  DcsAgent patching\n                 Status:  Success\n                Created:  November 4, 2020 5:02:53 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\ndcs-agent upgrade  to version 19.9.0.0.0 November 4, 2020 5:02:53 PM CET     November 4, 2020 5:04:28 PM CET     Success\nUpdate System version                    November 4, 2020 5:04:28 PM CET     November 4, 2020 5:04:28 PM CET     Success\n<\/pre>\n<p><\/br><br \/>\nWe will now update the DCS admin :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,13]\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli update-dcsadmin -v 19.9.0.0.0\n{\n  \"jobId\" : \"bdcbda55-d325-44ca-8bed-f0b15eeacfae\",\n  \"status\" : \"Created\",\n  \"message\" : null,\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 17:04:57 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"DcsAdmin patching\",\n  \"updatedTime\" : \"November 04, 2020 17:04:57 PM CET\"\n}\n\n[root@ODA01 patch]# odacli describe-job -i \"bdcbda55-d325-44ca-8bed-f0b15eeacfae\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  bdcbda55-d325-44ca-8bed-f0b15eeacfae\n            Description:  DcsAdmin patching\n                 Status:  Success\n                Created:  November 4, 2020 5:04:57 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nPatch location validation                November 4, 2020 5:04:58 PM CET     November 4, 2020 5:04:58 PM CET     Success\ndcs-admin upgrade                        November 4, 2020 5:04:58 PM CET     November 4, 2020 5:05:04 PM CET     Success\n<\/pre>\n<p><\/br><br \/>\nWe will update the DCS components :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1]\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli update-dcscomponents -v 19.9.0.0.0\n{\n  \"jobId\" : \"4782c035-86fd-496b-b9f1-1055d77071b3\",\n  \"status\" : \"Success\",\n  \"message\" : null,\n  \"reports\" : null,\n  \"createTimestamp\" : \"November 04, 2020 17:05:48 PM CET\",\n  \"description\" : \"Job completed and is not part of Agent job list\",\n  \"updatedTime\" : \"November 04, 2020 17:05:48 PM CET\"\n}\n<\/pre>\n<p><\/br><br \/>\nWe will run the prepatch report :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli create-prepatchreport -s -v 19.9.0.0.0\n\nJob details\n----------------------------------------------------------------\n                     ID:  d836f326-aba3-44e6-9be4-aaa031b5d730\n            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]\n                 Status:  Created\n                Created:  November 4, 2020 5:07:37 PM CET\n                Message:  Use 'odacli describe-prepatchreport -i d836f326-aba3-44e6-9be4-aaa031b5d730' to check details of results\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\n<\/pre>\n<p>And we will check the report :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,7,57,59]\">\n[root@ODA01 patch]# odacli describe-prepatchreport -i d836f326-aba3-44e6-9be4-aaa031b5d730\n\nPatch pre-check report\n------------------------------------------------------------------------\n                 Job ID:  d836f326-aba3-44e6-9be4-aaa031b5d730\n            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]\n                 Status:  FAILED\n                Created:  November 4, 2020 5:07:37 PM CET\n                 Result:  One or more pre-checks failed for [ORACHK]\n\nNode Name\n---------------\nODA01\n\nPre-Check                      Status   Comments\n------------------------------ -------- --------------------------------------\n__OS__\nValidate supported versions     Success   Validated minimum supported versions.\nValidate patching tag           Success   Validated patching tag: 19.9.0.0.0.\nIs patch location available     Success   Patch location is available.\nVerify OS patch                 Success   Verified OS patch\nValidate command execution      Success   Validated command execution\n\n__ILOM__\nValidate supported versions     Success   Validated minimum supported versions.\nValidate patching tag           Success   Validated patching tag: 19.9.0.0.0.\nIs patch location available     Success   Patch location is available.\nChecking Ilom patch Version     Success   Successfully verified the versions\nPatch location validation       Success   Successfully validated location\nValidate command execution      Success   Validated command execution\n\n__GI__\nValidate supported GI versions  Success   Validated minimum supported versions.\nValidate available space        Success   Validated free space under \/u01\nIs clusterware running          Success   Clusterware is running\nValidate patching tag           Success   Validated patching tag: 19.9.0.0.0.\nIs system provisioned           Success   Verified system is provisioned\nValidate ASM in online          Success   ASM is online\nValidate minimum agent version  Success   GI patching enabled in current\n                                          DCSAGENT version\nValidate GI patch metadata      Success   Validated patching tag: 19.9.0.0.0.\nValidate clones location exist  Success   Validated clones location\nIs patch location available     Success   Patch location is available.\nPatch location validation       Success   Successfully validated location\nPatch verification              Success   Patches 31771877 not applied on GI\n                                          home \/u01\/app\/19.0.0.0\/grid on node\n                                          ODA01\nValidate Opatch update          Success   Successfully updated the opatch in\n                                          GiHome \/u01\/app\/19.0.0.0\/grid on node\n                                          ODA01\nPatch conflict check            Success   No patch conflicts found on GiHome\n                                          \/u01\/app\/19.0.0.0\/grid on node\n                                          ODA01\nValidate command execution      Success   Validated command execution\n\n__ORACHK__\nRunning orachk                  Failed    Orachk validation failed: .\nValidate command execution      Success   Validated command execution\nSoftware home                   Failed    Software home check failed\n<\/pre>\n<p><\/br><br \/>\nThe prepatch report has been failing on orachk and the software home part. In the html report from orachk I could check and see that the software home check is failing due to missing files :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">\nFAIL =&gt; Software home check failed\n \nError Message:\nFile \"\/u01\/app\/19.0.0.0\/grid\/jdk\/jre\/lib\/amd64\/libjavafx_font_t2k.so\" could not be verified on node \"oda01\". OS error: \"No such file or directory\"\nError Message:\nFile \"\/u01\/app\/19.0.0.0\/grid\/jdk\/jre\/lib\/amd64\/libkcms.so\" could not be verified on node \"oda01\". OS error: \"No such file or directory\"\nError Message:\nFile \"\/u01\/app\/19.0.0.0\/grid\/rdbms\/lib\/ksms.o\" could not be verified on node \"oda01\". OS error: \"No such file or directory\"\n<\/pre>\n<p><\/br><br \/>\nThis files are expected during orachk check as referenced in the XML files :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,12]\">\n[root@ODA01 ~]# grep ksms \/u01\/app\/19.0.0.0\/grid\/cv\/cvdata\/ora_software_cfg.xml\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"ksms.o\" Permissions=\"644\"\/&gt;\n         &lt;File Path=\"bin\/\" Name=\"genksms\" Permissions=\"755\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"genksms.o\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"ksms.o\" Permissions=\"644\"\/&gt;\n         &lt;File Path=\"bin\/\" Name=\"genksms\" Permissions=\"755\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"genksms.o\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"ksms.o\" Permissions=\"644\"\/&gt;\n         &lt;File Path=\"bin\/\" Name=\"genksms\" Permissions=\"755\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"genksms.o\"\/&gt;\n\t\t \n[root@ODA01 ~]# grep ksms \/u01\/app\/19.0.0.0\/grid\/cv\/cvdata\/19\/ora_software_cfg.xml\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"ksms.o\" Permissions=\"644\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"genksms.o\"\/&gt;\n         &lt;File Path=\"bin\/\" Name=\"genksms\" Permissions=\"755\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"ksms.o\" Permissions=\"644\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"genksms.o\"\/&gt;\n         &lt;File Path=\"bin\/\" Name=\"genksms\" Permissions=\"755\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"ksms.o\" Permissions=\"644\"\/&gt;\n         &lt;File Path=\"rdbms\/lib\/\" Name=\"genksms.o\"\/&gt;\n         &lt;File Path=\"bin\/\" Name=\"genksms\" Permissions=\"755\"\/&gt;\n<\/pre>\n<p><\/br><br \/>\nI find following MOS note that can be related to same problem : File &#8220;$GRID_HOME\/rdbms\/lib\/ksms.o&#8221; could not be verified on node (Doc ID 1908505.1).<\/p>\n<p>As per this note we can ignore this erreur and move forward. I then decided to move forward with the server patching.<\/p>\n<p>I patched the server :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1]\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli update-server -v 19.9.0.0.0\n{\n  \"jobId\" : \"78f3ea84-4e31-4e1f-b195-eb4e75429102\",\n  \"status\" : \"Created\",\n  \"message\" : \"Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.\",\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 17:17:57 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Server Patching\",\n  \"updatedTime\" : \"November 04, 2020 17:17:57 PM CET\"\n}\n<\/pre>\n<p><\/br><br \/>\nBut the patching failed immediately as orachk was not successful due to the problem just described before :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1]\">\n[root@ODA01 patch]# odacli describe-job -i \"78f3ea84-4e31-4e1f-b195-eb4e75429102\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  78f3ea84-4e31-4e1f-b195-eb4e75429102\n            Description:  Server Patching\n                 Status:  Failure\n                Created:  November 4, 2020 5:17:57 PM CET\n                Message:  DCS-10702:Orachk validation failed: Please run describe-prepatchreport 78f3ea84-4e31-4e1f-b195-eb4e75429102 to see details.\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nServer patching                          November 4, 2020 5:18:05 PM CET     November 4, 2020 5:22:05 PM CET     Failure\nOrachk Server Patching                   November 4, 2020 5:18:05 PM CET     November 4, 2020 5:22:05 PM CET     Failure\n<\/pre>\n<p><\/br><br \/>\nSo starting 19.9 it seems that orachk is mandatory before doing any patching, and if orachk will not be successful the patching will then fail.<\/p>\n<p>By chance there is a new skip-orachk option to skip the orachk during server patching :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,19,20]\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli update-server -v 19.9.0.0.0 -h\nUsage: update-server [options]\n  Options:\n    --component, -c\n      The component that is requested for update. The supported components\n      include: OS\n    --force, -f\n      Ignore precheck error and force patching\n    --help, -h\n      get help\n    --json, -j\n      json output\n    --local, -l\n      Update Server Components Locally\n    --node, -n\n      Node to be updated\n    --precheck, -p\n      Obsolete flag\n    --skip-orachk, -sko\n      Option to skip orachk validations\n    --version, -v\n      Version to be updated\n<\/pre>\n<p><\/br><br \/>\nI then could successfully patch the server :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1]\">\n[root@ODA01 patch]# \/opt\/oracle\/dcs\/bin\/odacli update-server -v 19.9.0.0.0 -sko\n{\n  \"jobId\" : \"878fac12-a2a0-4302-955c-7df3d4fdd517\",\n  \"status\" : \"Created\",\n  \"message\" : \"Success of server update will trigger reboot of the node after 4-5 minutes. Please wait until the node reboots.\",\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 18:03:15 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Server Patching\",\n  \"updatedTime\" : \"November 04, 2020 18:03:15 PM CET\"\n}\n<\/pre>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,4,10]\">\n[root@ODA01 ~]# uptime\n 19:06:00 up 2 min,  1 user,  load average: 2.58, 1.32, 0.52\n \n[root@ODA01 ~]# odacli describe-job -i \"878fac12-a2a0-4302-955c-7df3d4fdd517\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  878fac12-a2a0-4302-955c-7df3d4fdd517\n            Description:  Server Patching\n                 Status:  Success\n                Created:  November 4, 2020 6:03:15 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nPatch location validation                November 4, 2020 6:03:23 PM CET     November 4, 2020 6:03:23 PM CET     Success\ndcs-controller upgrade                   November 4, 2020 6:03:23 PM CET     November 4, 2020 6:03:28 PM CET     Success\nPatch location validation                November 4, 2020 6:03:30 PM CET     November 4, 2020 6:03:30 PM CET     Success\ndcs-cli upgrade                          November 4, 2020 6:03:30 PM CET     November 4, 2020 6:03:30 PM CET     Success\nCreating repositories using yum          November 4, 2020 6:03:30 PM CET     November 4, 2020 6:03:33 PM CET     Success\nUpdating YumPluginVersionLock rpm        November 4, 2020 6:03:33 PM CET     November 4, 2020 6:03:33 PM CET     Success\nApplying OS Patches                      November 4, 2020 6:03:33 PM CET     November 4, 2020 6:13:18 PM CET     Success\nCreating repositories using yum          November 4, 2020 6:13:18 PM CET     November 4, 2020 6:13:18 PM CET     Success\nApplying HMP Patches                     November 4, 2020 6:13:18 PM CET     November 4, 2020 6:13:38 PM CET     Success\nClient root Set up                       November 4, 2020 6:13:38 PM CET     November 4, 2020 6:13:41 PM CET     Success\nClient grid Set up                       November 4, 2020 6:13:41 PM CET     November 4, 2020 6:13:46 PM CET     Success\nPatch location validation                November 4, 2020 6:13:46 PM CET     November 4, 2020 6:13:46 PM CET     Success\noda-hw-mgmt upgrade                      November 4, 2020 6:13:46 PM CET     November 4, 2020 6:14:17 PM CET     Success\nOSS Patching                             November 4, 2020 6:14:17 PM CET     November 4, 2020 6:14:18 PM CET     Success\nApplying Firmware Disk Patches           November 4, 2020 6:14:18 PM CET     November 4, 2020 6:14:21 PM CET     Success\nApplying Firmware Controller Patches     November 4, 2020 6:14:21 PM CET     November 4, 2020 6:14:24 PM CET     Success\nChecking Ilom patch Version              November 4, 2020 6:14:25 PM CET     November 4, 2020 6:14:27 PM CET     Success\nPatch location validation                November 4, 2020 6:14:27 PM CET     November 4, 2020 6:14:28 PM CET     Success\nSave password in Wallet                  November 4, 2020 6:14:29 PM CET     November 4, 2020 6:14:30 PM CET     Success\nApply Ilom patch                         November 4, 2020 6:14:30 PM CET     November 4, 2020 6:22:34 PM CET     Success\nCopying Flash Bios to Temp location      November 4, 2020 6:22:34 PM CET     November 4, 2020 6:22:34 PM CET     Success\nStarting the clusterware                 November 4, 2020 6:22:35 PM CET     November 4, 2020 6:23:58 PM CET     Success\nclusterware patch verification           November 4, 2020 6:23:58 PM CET     November 4, 2020 6:24:01 PM CET     Success\nPatch location validation                November 4, 2020 6:24:01 PM CET     November 4, 2020 6:24:01 PM CET     Success\nOpatch update                            November 4, 2020 6:24:43 PM CET     November 4, 2020 6:24:46 PM CET     Success\nPatch conflict check                     November 4, 2020 6:24:46 PM CET     November 4, 2020 6:25:31 PM CET     Success\nclusterware upgrade                      November 4, 2020 6:25:52 PM CET     November 4, 2020 6:50:57 PM CET     Success\nUpdating GiHome version                  November 4, 2020 6:50:57 PM CET     November 4, 2020 6:51:12 PM CET     Success\nUpdate System version                    November 4, 2020 6:51:16 PM CET     November 4, 2020 6:51:16 PM CET     Success\nCleanup JRE Home                         November 4, 2020 6:51:16 PM CET     November 4, 2020 6:51:16 PM CET     Success\npreRebootNode Actions                    November 4, 2020 6:51:16 PM CET     November 4, 2020 6:51:57 PM CET     Success\nReboot Ilom                              November 4, 2020 6:51:57 PM CET     November 4, 2020 6:51:57 PM CET     Success\n<\/pre>\n<p><\/br><br \/>\nI could check the new current installed version :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1]\">\n[root@ODA01 ~]# odacli describe-component\nSystem Version\n---------------\n19.9.0.0.0\n\nComponent                                Installed Version    Available Version\n---------------------------------------- -------------------- --------------------\nOAK                                       19.9.0.0.0            up-to-date\nGI                                        19.9.0.0.201020       up-to-date\nDB                                        18.7.0.0.190716       18.12.0.0.201020\nDCSAGENT                                  19.9.0.0.0            up-to-date\nILOM                                      5.0.1.21.r136383      up-to-date\nBIOS                                      52030400              up-to-date\nOS                                        7.8                   up-to-date\nFIRMWARECONTROLLER                        VDV1RL02              VDV1RL04\nFIRMWAREDISK                              1102                  1132\nHMP                                       2.4.7.0.1             up-to-date\n<\/pre>\n<p><\/br><br \/>\nI patched the storage :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,13]\">\n[root@ODA01 ~]# odacli update-storage -v 19.9.0.0.0\n{\n  \"jobId\" : \"61871e3d-088b-43af-8b91-94dc4fa1331a\",\n  \"status\" : \"Created\",\n  \"message\" : \"Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart\",\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 19:07:17 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Storage Firmware Patching\",\n  \"updatedTime\" : \"November 04, 2020 19:07:17 PM CET\"\n}\n\n[root@ODA01 ~]# odacli describe-job -i \"61871e3d-088b-43af-8b91-94dc4fa1331a\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  61871e3d-088b-43af-8b91-94dc4fa1331a\n            Description:  Storage Firmware Patching\n                 Status:  Success\n                Created:  November 4, 2020 7:07:17 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nApplying Firmware Disk Patches           November 4, 2020 7:07:20 PM CET     November 4, 2020 7:07:21 PM CET     Success\npreRebootNode Actions                    November 4, 2020 7:07:21 PM CET     November 4, 2020 7:07:21 PM CET     Success\nReboot Ilom                              November 4, 2020 7:07:21 PM CET     November 4, 2020 7:07:21 PM CET     Success\n<\/pre>\n<p><\/br><br \/>\nSurprisingly the storage patching was done immediately and with no reboot.<\/p>\n<p>I checked the version and could see that effectively the storage was still running old firmware versions :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,15,16]\">\n[root@ODA01 ~]# odacli describe-component\nSystem Version\n---------------\n19.9.0.0.0\n\nComponent                                Installed Version    Available Version\n---------------------------------------- -------------------- --------------------\nOAK                                       19.9.0.0.0            up-to-date\nGI                                        19.9.0.0.201020       up-to-date\nDB                                        18.7.0.0.190716       18.12.0.0.201020\nDCSAGENT                                  19.9.0.0.0            up-to-date\nILOM                                      5.0.1.21.r136383      up-to-date\nBIOS                                      52030400              up-to-date\nOS                                        7.8                   up-to-date\nFIRMWARECONTROLLER                        VDV1RL02              VDV1RL04\nFIRMWAREDISK                              1102                  1132\nHMP                                       2.4.7.0.1             up-to-date\n<\/pre>\n<p><\/br><br \/>\nI opened a SR and could get confirmation from support that this is a bug. It will not impact any functionnality.<br \/>\nThe BUG is the following one :<br \/>\nBug 32017186 &#8211; LNX64-199-CMT : FIRMWARECONTROLLER NOT PATCHED FOR 19.9<\/p>\n<p><strong>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;- UPDATE FROM 12.11.2020 &#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-<\/strong><br \/>\nOracle support gave me the WA to solve firmware disk and controller patching issue.<\/p>\n<p>Firmware disk version 1132 is not yet available, thus the xml metadata file should include version 1102.<\/p>\n<p>We will then backup the current \/opt\/oracle\/oak\/pkgrepos\/System\/latest\/patchmetadata.xml file :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">\n[root@ODA01 ~]# cp -p \/opt\/oracle\/oak\/pkgrepos\/System\/latest\/patchmetadata.xml \/opt\/oracle\/oak\/pkgrepos\/System\/latest\/patchmetadata.xml.orig.20201109\n<\/pre>\n<p><\/br><\/p>\n<p>And we will transfer the new file provided by oracle support (<a href=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2020\/11\/19.9.patchmetadata.xml_.zip\">19.9.patchmetadata.xml<\/a>) :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,3,7,10]\">\n[root@ODA01 ~]# cd  \/opt\/oracle\/oak\/pkgrepos\/System\/latest\n\n[root@ODA01 latest]# ls -ltrh *patchmetadata.xml\n-rwx------ 1 root root 39K Oct 29 05:10 patchmetadata.xml\n-rw-r--r-- 1 root root 39K Nov  9 16:02 19.9.patchmetadata.xml\n\n[root@ODA01 latest]#  mv 19.9.patchmetadata.xml patchmetadata.xml\nmv: overwrite \u2018patchmetadata.xml\u2019? y\n\n[root@ODA01 latest]# diff patchmetadata.xml patchmetadata.xml.orig.20201109\n1019c1019\n&lt;       1102,1132\n---\n&gt;       1132\n[root@ODA01 latest]#\n<\/pre>\n<p><\/br><\/p>\n<p>The odacli describe-component command is now showing the correct output for firmwaredisk :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,16]\">\n[root@ODA01 latest]# odacli describe-component\nSystem Version\n---------------\n19.9.0.0.0\n\nComponent                                Installed Version    Available Version\n---------------------------------------- -------------------- --------------------\nOAK                                       19.9.0.0.0            up-to-date\nGI                                        19.9.0.0.201020       up-to-date\nDB                                        18.7.0.0.190716       18.12.0.0.201020\nDCSAGENT                                  19.9.0.0.0            up-to-date\nILOM                                      5.0.1.21.r136383      up-to-date\nBIOS                                      52030400              up-to-date\nOS                                        7.8                   up-to-date\nFIRMWARECONTROLLER                        VDV1RL02              VDV1RL04\nFIRMWAREDISK                              1102                  up-to-date\nHMP                                       2.4.7.0.1             up-to-date\n<\/pre>\n<p><\/br><\/p>\n<p>As per the firmwarecontroller we will patch it manually as following. There is no impact and downtime. No reboot is needed.<\/p>\n<p>The firmware to be patched manually is the VDV1RL04 version and stored in :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,3]\">\n[root@ODA01 log]# cd \/opt\/oracle\/oak\/pkgrepos\/firmwarecontroller\/intel\/0x0a54\/vdv1rl04\/7361456_icrpc2dd2ora6.4t\n\n[root@ODA01 7361456_icrpc2dd2ora6.4t]# ls\ncomponentmetadata.xml  ICRPC2DD2.RL04.fw  metadata.xml\n<\/pre>\n<p><\/br><\/p>\n<p>We will check the current NVMe Firmware controller disk version. It should be VDV1RL02 for all NVMe disks :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,8,9,10,11,12,13]\">\n[root@ODA01 7361456_icrpc2dd2ora6.4t]#  fwupdate list controller\n\n==================================================\nCONTROLLER\n==================================================\nID    Type   Manufacturer   Model     Product Name              FW Version     BIOS Version   EFI Version    FCODE Version  Package Version  NVDATA Version    XML Support\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nc0    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL02       -              -              -              -                -                 N\/A\nc1    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL02       -              -              -              -                -                 N\/A\nc2    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL02       -              -              -              -                -                 N\/A\nc3    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL02       -              -              -              -                -                 N\/A\nc4    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL02       -              -              -              -                -                 N\/A\nc5    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL02       -              -              -              -                -                 N\/A\nc6    HDC    Intel          0x2826    0x486c                    -              -              -              -              -                -                 N\/A\nc7    NET    Intel          0x1533    Intel(R) I210 Gigabit Net -              -                                            80000681         -                 N\/A\n<\/pre>\n<p><\/br><\/p>\n<p>We will patch them manually. Following command needs to be run on all NVME disks ID, in my case c0, c1, c2, c3, c4 and c5. Example of command is given for ID c1 :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1\">\n[root@ODA01 7361456_icrpc2dd2ora6.4t]# fwupdate update controller -n c1 -x metadata.xml\n\nThe following actions will be taken:\n==========================================================\nID        Priority Action     Status      Old Firmware Ver.   Proposed Ver.       New Firmware Ver.   System Reboot\n------------------------------------------------------------------------------------------------------------------------\nc1        1        Check FW   Success     VDV1RL02            VDV1RL04            N\/A                 None\nDo you wish to process the above actions? [y\/n]? y\nUpdating c1:  Success\nSleeping for 10 seconds for component to recover\nResetting c1\nMandatory post reset 60 second sleep\n\nVerifying all priority 1 updates\n\nExecution Summary\n==========================================================\nID        Priority Action     Status      Old Firmware Ver.   Proposed Ver.       New Firmware Ver.   System Reboot\n------------------------------------------------------------------------------------------------------------------------\nc1        1        Validate   Success     VDV1RL02            VDV1RL04            VDV1RL04            None\n<\/pre>\n<p><\/br><\/p>\n<p>Now all NVMe disks are showing a controller version of VDV1RL04 :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,8,9,10,11,12,13]\">\n[root@ODA01 7361456_icrpc2dd2ora6.4t]# fwupdate list controller\n\n==================================================\nCONTROLLER\n==================================================\nID    Type   Manufacturer   Model     Product Name              FW Version     BIOS Version   EFI Version    FCODE Version  Package Version  NVDATA Version    XML Support\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------\nc0    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL04       -              -              -              -                -                 N\/A\nc1    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL04       -              -              -              -                -                 N\/A\nc2    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL04       -              -              -              -                -                 N\/A\nc3    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL04       -              -              -              -                -                 N\/A\nc4    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL04       -              -              -              -                -                 N\/A\nc5    NVMe   Intel          0x0a54    7361456_ICRPC2DD2ORA6.4T  VDV1RL04       -              -              -              -                -                 N\/A\nc6    HDC    Intel          0x2826    0x486c                    -              -              -              -              -                -                 N\/A\nc7    NET    Intel          0x1533    Intel(R) I210 Gigabit Net -              -                                            80000681         -                 N\/A\n<\/pre>\n<p><\/br><\/p>\n<p>The command odacli describe-component is now showing firmware controller to be updated :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,18]\">\n[root@ODA01 7361456_icrpc2dd2ora6.4t]# odacli describe-component\nSystem Version\n---------------\n19.9.0.0.0\n\nComponent                                Installed Version    Available Version\n---------------------------------------- -------------------- --------------------\nOAK                                       19.9.0.0.0            up-to-date\nGI                                        19.9.0.0.201020       up-to-date\nDB {\n[ OraDB18000_home1,OraDB18000_home2 ]     18.7.0.0.190716       18.12.0.0.201020\n[ OraDB19000_home1,OraDB19000_home2 ]     19.9.0.0.201020       up-to-date\n}\nDCSAGENT                                  19.9.0.0.0            up-to-date\nILOM                                      5.0.1.21.r136383      up-to-date\nBIOS                                      52030400              up-to-date\nOS                                        7.8                   up-to-date\nFIRMWARECONTROLLER                        VDV1RL04              up-to-date\nFIRMWAREDISK                              1102                  up-to-date\nHMP                                       2.4.7.0.1             up-to-date\n\n<\/pre>\n<p><\/br><br \/>\n<strong>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8211;<\/strong><\/p>\n<p>As per the rdbms home, they can be patched later. If we are using a High Availability solution, both primary and standby databases&#8217; homes need to be patched during same maintenance windows.<\/p>\n<h3>Post patching activities<\/h3>\n<p>We can now run post patching activities.<\/p>\n<p>We will ensure there is no new hardware problem :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [16]\">\nlogin as: root\nKeyboard-interactive authentication prompts from server:\n| Password:\nEnd of keyboard-interactive prompts from server\n\nOracle(R) Integrated Lights Out Manager\n\nVersion 5.0.1.21 r136383\n\nCopyright (c) 2020, Oracle and\/or its affiliates. All rights reserved.\n\nWarning: HTTPS certificate is set to factory default.\n\nHostname: ODA01-ILOM\n\n-&gt; show \/SP\/faultmgmt\n\n \/SP\/faultmgmt\n    Targets:\n        shell\n\n    Properties:\n\n    Commands:\n        cd\n        show\n\n-&gt;\n<\/pre>\n<p><\/br><br \/>\nWe can also remove our odabr snapshot backup :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,3,21]\">\n[root@ODA01 ~]# export PATH=\/opt\/odabr:$PATH\n\n[root@ODA01 ~]# odabr infosnap\n\n--------------------------------------------------------\nodabr - ODA node Backup Restore\nAuthor: Ruggero Citton \nRAC Pack, Cloud Innovation and Solution Engineering Team\nCopyright Oracle, Inc. 2013, 2019\nVersion: 2.0.1-47\n--------------------------------------------------------\n\n\nLVM snap name         Status                COW Size              Data%\n-------------         ----------            ----------            ------\nroot_snap             active                30.00 GiB             22.79%\nopt_snap              active                60.00 GiB             34.37%\nu01_snap              active                100.00 GiB            35.58%\n\n\n[root@ODA01 ~]# odabr delsnap\nINFO: 2020-11-04 19:31:46: Please check the logfile '\/opt\/odabr\/out\/log\/odabr_81687.log' for more details\n\nINFO: 2020-11-04 19:31:46: Removing LVM snapshots\nINFO: 2020-11-04 19:31:46: ...removing LVM snapshot for 'opt'\nSUCCESS: 2020-11-04 19:31:46: ...snapshot for 'opt' removed successfully\nINFO: 2020-11-04 19:31:46: ...removing LVM snapshot for 'u01'\nSUCCESS: 2020-11-04 19:31:47: ...snapshot for 'u01' removed successfully\nINFO: 2020-11-04 19:31:47: ...removing LVM snapshot for 'root'\nSUCCESS: 2020-11-04 19:31:47: ...snapshot for 'root' removed successfully\nSUCCESS: 2020-11-04 19:31:47: Remove LVM snapshots done successfully\n<\/pre>\n<p><\/br><br \/>\nWe can cleanup previous patching version from repository and give additionnal space to \/opt :<\/p>\n<pre class=\"brush: sql; gutter: true; first-line: 1; highlight: [1,5,7,19,34,38]\">\n[root@ODA01 ~]# df -h \/ \/u01 \/opt\nFilesystem                          Size  Used Avail Use% Mounted on\n\/dev\/mapper\/VolGroupSys-LogVolRoot   30G   11G   18G  38% \/\n\/dev\/mapper\/VolGroupSys-LogVolU01    99G   59G   35G  63% \/u01\n\/dev\/mapper\/VolGroupSys-LogVolOpt    75G   60G   12G  84% \/opt\n\n[root@ODA01 ~]# odacli cleanup-patchrepo -comp GI,DB -v 19.6.0.0.0\n{\n  \"jobId\" : \"97b9669b-6945-4358-938e-a3a3f3b73693\",\n  \"status\" : \"Created\",\n  \"message\" : null,\n  \"reports\" : [ ],\n  \"createTimestamp\" : \"November 04, 2020 19:32:16 PM CET\",\n  \"resourceList\" : [ ],\n  \"description\" : \"Cleanup patchrepos\",\n  \"updatedTime\" : \"November 04, 2020 19:32:16 PM CET\"\n}\n\n[root@ODA01 ~]# odacli describe-job -i \"97b9669b-6945-4358-938e-a3a3f3b73693\"\n\nJob details\n----------------------------------------------------------------\n                     ID:  97b9669b-6945-4358-938e-a3a3f3b73693\n            Description:  Cleanup patchrepos\n                 Status:  Success\n                Created:  November 4, 2020 7:32:16 PM CET\n                Message:\n\nTask Name                                Start Time                          End Time                            Status\n---------------------------------------- ----------------------------------- ----------------------------------- ----------\nCleanup Repository                       November 4, 2020 7:32:17 PM CET     November 4, 2020 7:32:17 PM CET     Success\nCleanup JRE Home                         November 4, 2020 7:32:17 PM CET     November 4, 2020 7:32:17 PM CET     Success\n\n[root@ODA01 ~]# df -h \/ \/u01 \/opt\nFilesystem                          Size  Used Avail Use% Mounted on\n\/dev\/mapper\/VolGroupSys-LogVolRoot   30G   11G   18G  38% \/\n\/dev\/mapper\/VolGroupSys-LogVolU01    99G   59G   35G  63% \/u01\n\/dev\/mapper\/VolGroupSys-LogVolOpt    75G   49G   23G  68% \/opt\n<\/pre>\n<p>We can restart our databases with srvctl start database command or srvctl start home command.<\/p>\n<p>Finally we will activate our database synchronization if using Data Guard or dbvisit.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>ODA 19.9 has just been released for Bare Metal yesterday, and I had the opportunity to already patch a customer production ODA to this latest version. Through this blog I wanted to share my experience on patching an ODA to 19.9 as well as a new tricky skip-orachk option.<\/p>\n","protected":false},"author":48,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229,59],"tags":[2145,79,155,2168],"type_dbi":[],"class_list":["post-15062","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","category-oracle","tag-19-9","tag-oda","tag-patching","tag-skip-orachk"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Patching Oracle Database Appliance to 19.9 - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Patching Oracle Database Appliance to 19.9\" \/>\n<meta property=\"og:description\" content=\"ODA 19.9 has just been released for Bare Metal yesterday, and I had the opportunity to already patch a customer production ODA to this latest version. Through this blog I wanted to share my experience on patching an ODA to 19.9 as well as a new tricky skip-orachk option.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2020-11-05T08:35:27+00:00\" \/>\n<meta name=\"author\" content=\"Marc Wagner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marc Wagner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"25 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\"},\"author\":{\"name\":\"Marc Wagner\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\"},\"headline\":\"Patching Oracle Database Appliance to 19.9\",\"datePublished\":\"2020-11-05T08:35:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\"},\"wordCount\":901,\"commentCount\":0,\"keywords\":[\"19.9\",\"ODA\",\"Patching\",\"skip-orachk\"],\"articleSection\":[\"Database Administration &amp; Monitoring\",\"Oracle\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\",\"name\":\"Patching Oracle Database Appliance to 19.9 - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2020-11-05T08:35:27+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Patching Oracle Database Appliance to 19.9\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628\",\"name\":\"Marc Wagner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g\",\"caption\":\"Marc Wagner\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/marc-wagner\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Patching Oracle Database Appliance to 19.9 - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/","og_locale":"en_US","og_type":"article","og_title":"Patching Oracle Database Appliance to 19.9","og_description":"ODA 19.9 has just been released for Bare Metal yesterday, and I had the opportunity to already patch a customer production ODA to this latest version. Through this blog I wanted to share my experience on patching an ODA to 19.9 as well as a new tricky skip-orachk option.","og_url":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/","og_site_name":"dbi Blog","article_published_time":"2020-11-05T08:35:27+00:00","author":"Marc Wagner","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Marc Wagner","Est. reading time":"25 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/"},"author":{"name":"Marc Wagner","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628"},"headline":"Patching Oracle Database Appliance to 19.9","datePublished":"2020-11-05T08:35:27+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/"},"wordCount":901,"commentCount":0,"keywords":["19.9","ODA","Patching","skip-orachk"],"articleSection":["Database Administration &amp; Monitoring","Oracle"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/","url":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/","name":"Patching Oracle Database Appliance to 19.9 - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2020-11-05T08:35:27+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/patching-oracle-database-appliance-to-19-9\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Patching Oracle Database Appliance to 19.9"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/225d9884b8467ead9a872823acb14628","name":"Marc Wagner","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a873cc6e7fbdbbcbdbcaf5dbded14ad9a77b2ec2c3e03b4d724ed33d35d5f328?s=96&d=mm&r=g","caption":"Marc Wagner"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/marc-wagner\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/15062","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/48"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=15062"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/15062\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=15062"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=15062"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=15062"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=15062"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}