Here is the latest patch for your ODA and it seems that Oracle documentation for this patch is not exactly the procedure you’ll have to follow to successfully patch your appliance. I recently updated X6-2M and X6-2L to this latest release and here is how to do that. In this example I was patching from 12.1.2.12.0, no intermediate patch was needed.
1) Download the patch
It seems odd but sometimes finding the corresponding patch is not so easy! With the patch number, it’s more convenient. For 12.2.1.3.0 with the dcs stack the number is 27648057. This patch will update all the components: dcs (odacli), operating system, bios/firmwares, ilom, GI and dbhomes.
Copy the patch in a temporary folder on the server, for example /opt/patch. You’ll need to be root to apply the patch.
2) Check the actual versions and free up space on disk
I recommend you to check the actual versions. It’s easy:
odacli describe-component System Version --------------- 12.1.2.12.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 12.1.2.12.0 up-to-date GI 12.1.0.2.170814 up-to-date DB { [ OraDB12102_home1 ] 12.1.0.2.170814 up-to-date [ OraDB11204_home1 ] 11.2.0.4.170814 up-to-date } ILOM 3.2.7.26.a.r112632 up-to-date BIOS 39050100 up-to-date OS 6.8 up-to-date FIRMWARECONTROLLER { [ c6 ] 4.650.00-7176 up-to-date [ c0,c1,c2,c3,c4,c5 KPYABR3Q up-to-date } FIRMWAREDISK 0R3Q up-to-date
Don’t care about Available Version column, it’s only valid after you register a new patch. Your ODA doesn’t check the latest patch online.
For free space check that folders /, /u01 and /opt have enough free GB to process. At least 10GB I think.
3) Prepare the patch files
It’s slightly different than previous versions. Only the first zipfile has to be uncompressed and registered. The 2 other files can directly be registered without unzipping them.
cd /opt/patch unzip p27648057_122130_Linux-x86-64_1of3.zip odacli update-repository -f /opt/patch/oda-sm-12.2.1.3.0-180504-server1of3.zip odacli update-repository -f /opt/patch/p27648057_122130_Linux-x86-64_2of3.zip odacli update-repository -f /opt/patch/p27648057_122130_Linux-x86-64_3of3.zip
4) Update the dcs-agent
First you’ll have to update the dcs-agent:
odacli update-dcsagent -v 12.2.1.3.0 { "jobId" : "150b3486-cfb2-4b32-b751-0ed89ce3d7be", "status" : "Created", "message" : "Dcs agent will be restarted after the update. Please wait for 2-3 mins before executing the other commands", "reports" : [ ], "createTimestamp" : "May 22, 2018 10:31:50 AM CEST", "resourceList" : [ ], "description" : "DcsAgent patching", "updatedTime" : "May 22, 2018 10:31:50 AM CEST" }
As for every kind of operation you do with odacli, you receive a jobId you can monitor:
odacli describe-job -i "150b3486-cfb2-4b32-b751-0ed89ce3d7be" Job details ---------------------------------------------------------------- ID: 150b3486-cfb2-4b32-b751-0ed89ce3d7be Description: DcsAgent patching Status: Success Created: May 22, 2018 10:31:50 AM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- dcsagent rpm verification May 22, 2018 10:31:50 AM CEST May 22, 2018 10:31:50 AM CEST Success Patch location validation May 22, 2018 10:31:50 AM CEST May 22, 2018 10:31:50 AM CEST Success dcs-agent upgrade May 22, 2018 10:31:50 AM CEST May 22, 2018 10:31:51 AM CEST Success
This update takes only a minute.
Check again the version and you will see a new component in 18c, quite weird:
odacli describe-component System Version --------------- 12.2.1.3.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 12.1.2.12.0 12.2.1.3.0 GI 12.1.0.2.170814 12.2.0.1.180116 DB { [ OraDB12102_home1 ] 12.1.0.2.170814 12.1.0.2.180116 [ OraDB11204_home1 ] 11.2.0.4.170814 11.2.0.4.180116 } DCSAGENT 18.1.3.0.0 up-to-date ILOM 3.2.7.26.a.r112632 up-to-date BIOS 39050100 39090000 OS 6.8 6.9 FIRMWARECONTROLLER { [ c6 ] 4.650.00-7176 up-to-date [ c0,c1,c2,c3,c4,c5 ] KPYABR3Q kpyagr3q } FIRMWAREDISK 0R3Q up-to-date
5) Update the server
Updating the server will actually update not only the operating system but also the ILOM, the BIOS, the firmware of internal disks and the GI. For the OS, Oracle Linux will go from 6.8 to 6.9 and the update process will assume that no additional packages have been installed. If you installed additional packages, please remove them as they can prevent the patch to apply correctly. Even devel packages should be removed, the upgrade of normal packages linked to them will not work because of the dependencies. You can easily remove the install packages with rpm -e, for example:
rpm -e openssl-devel-1.0.1e-48.el6_8.4.x86_64 rpm -e krb5-devel-1.10.3-57.el6.x86_64 rpm -e zlib-devel-1.2.3-29.el6.x86_64 rpm -e keyutils-libs-devel-1.4-5.0.1.el6.x86_64 rpm -e libcom_err-devel-1.42.8-1.0.2.el6.x86_64 rpm -e libselinux-devel-2.0.94-7.el6.x86_64 rpm -e libsepol-devel-2.0.41-4.el6.x86_64
Now you can safely run the patching:
odacli update-server -v 12.2.1.3.0
This update is the longest one (between 30 minutes and 1 hour), while you should see your server rebooting. As usual check the status of the job with describe-job:
odacli describe-job -i "27d2195f-f16b-44d8-84e0-6af6e48ccad7"
At the end of the process, describe-job will look like that:
Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Patch location validation May 23, 2018 11:07:09 AM CEST May 23, 2018 11:07:09 AM CEST Success dcs-controller upgrade May 23, 2018 11:07:09 AM CEST May 23, 2018 11:07:09 AM CEST Success Patch location validation May 23, 2018 11:07:09 AM CEST May 23, 2018 11:07:09 AM CEST Success dcs-cli upgrade May 23, 2018 11:07:09 AM CEST May 23, 2018 11:07:09 AM CEST Success Creating repositories using yum May 23, 2018 11:07:09 AM CEST May 23, 2018 11:07:10 AM CEST Success Applying HMP Patches May 23, 2018 11:07:10 AM CEST May 23, 2018 11:09:04 AM CEST Success Patch location validation May 23, 2018 11:09:04 AM CEST May 23, 2018 11:09:04 AM CEST Success oda-hw-mgmt upgrade May 23, 2018 11:09:04 AM CEST May 23, 2018 11:09:04 AM CEST Success Creating repositories using yum May 23, 2018 11:09:04 AM CEST May 23, 2018 11:09:09 AM CEST Success Applying OS Patches May 23, 2018 11:09:09 AM CEST May 23, 2018 11:11:04 AM CEST Success OSS Patching May 23, 2018 11:11:04 AM CEST May 23, 2018 11:11:04 AM CEST Success Applying Firmware Disk Patches May 23, 2018 11:11:04 AM CEST May 23, 2018 11:11:08 AM CEST Success Applying Firmware Expander Patches May 23, 2018 11:11:08 AM CEST May 23, 2018 11:11:13 AM CEST Success Applying Firmware Controller Patches May 23, 2018 11:11:13 AM CEST May 23, 2018 11:11:16 AM CEST Success Checking Ilom patch Version May 23, 2018 11:11:17 AM CEST May 23, 2018 11:11:19 AM CEST Success Patch location validation May 23, 2018 11:11:19 AM CEST May 23, 2018 11:11:20 AM CEST Success Apply Ilom patch May 23, 2018 11:11:21 AM CEST May 23, 2018 11:11:22 AM CEST Success Copying Flash Bios to Temp location May 23, 2018 11:11:22 AM CEST May 23, 2018 11:11:22 AM CEST Success Starting the clusterware May 23, 2018 11:11:23 AM CEST May 23, 2018 11:11:23 AM CEST Success Creating GI home directories May 23, 2018 11:11:23 AM CEST May 23, 2018 11:11:23 AM CEST Success Cloning Gi home May 23, 2018 11:11:23 AM CEST May 23, 2018 11:13:53 AM CEST Success Configuring GI May 23, 2018 11:13:53 AM CEST May 23, 2018 11:14:06 AM CEST Success Running GI upgrade root scripts May 23, 2018 11:14:06 AM CEST May 23, 2018 11:29:04 AM CEST Success Resetting DG compatibility May 23, 2018 11:29:04 AM CEST May 23, 2018 11:29:09 AM CEST Success Running GI config assistants May 23, 2018 11:29:09 AM CEST May 23, 2018 11:30:10 AM CEST Success restart oakd May 23, 2018 11:30:13 AM CEST May 23, 2018 11:30:23 AM CEST Success Updating GiHome version May 23, 2018 11:30:23 AM CEST May 23, 2018 11:30:25 AM CEST Success preRebootNode Actions May 23, 2018 11:30:39 AM CEST May 23, 2018 11:31:25 AM CEST Success Reboot Ilom May 23, 2018 11:31:25 AM CEST May 23, 2018 11:31:25 AM CEST Success
Don’t forget that if there is a problem during the patching process (you forgot to remove an additional rpm for example), you can relaunch the patching and it will skip the already patched components. But you will loose a lot of time! Please control the components version after the reboot with describe-component.
6) Patch the dbhomes
You now need to patch the dbhomes separately. First of all list them:
odacli list-dbhomes ID Name DB Version Home Location Status ---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ---------- 86b6a068-55a8-4171-9f94-48b86f135065 OraDB12102_home1 12.1.0.2.170814 (26680878, 26609798) /u01/app/oracle/product/12.1.0.2/dbhome_1 Configured c6e7d556-5785-41d8-a120-ed3ef756848a OraDB11204_home1 11.2.0.4.170814 (26609929, 26609445) /u01/app/oracle/product/11.2.0.4/dbhome_1 Configured
And then patch the first one:
odacli update-dbhome -v 12.2.1.3 -i c6e7d556-5785-41d8-a120-ed3ef756848a
It will also apply the datapatch on all the primary databases connected to this home. If you don’t have any database linked to the dbhome patching is fast:
odacli describe-job -i "9defa52d-2997-42ea-9bd3-aafdbf0a4dc5" Job details ---------------------------------------------------------------- ID: 9defa52d-2997-42ea-9bd3-aafdbf0a4dc5 Description: DB Home Patching: Home Id is c6e7d556-5785-41d8-a120-ed3ef756848a Status: Success Created: May 23, 2018 11:43:55 AM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- clusterware patch verification May 23, 2018 11:43:56 AM CEST May 23, 2018 11:43:58 AM CEST Success Patch location validation May 23, 2018 11:43:58 AM CEST May 23, 2018 11:43:59 AM CEST Success Opatch updation May 23, 2018 11:44:19 AM CEST May 23, 2018 11:44:20 AM CEST Success Patch conflict check May 23, 2018 11:44:20 AM CEST May 23, 2018 11:44:27 AM CEST Success db upgrade May 23, 2018 11:44:27 AM CEST May 23, 2018 11:46:01 AM CEST Success
About 2 minutes for an empty dbhome.
And for a dbhome linked to databases:
odacli update-dbhome -v 12.2.1.3 -i 86b6a068-55a8-4171-9f94-48b86f135065 { "jobId" : "8083dc43-61fe-49da-8081-43b4e5257e95", "status" : "Created", "message" : null, "reports" : [ ], "createTimestamp" : "May 23, 2018 12:46:06 PM CEST", "resourceList" : [ ], "description" : "DB Home Patching: Home Id is 86b6a068-55a8-4171-9f94-48b86f135065", "updatedTime" : "May 23, 2018 12:46:06 PM CEST" } odacli describe-job -i "e871f741-7dd6-49c6-9b4a-af0d68e647e7" Job details ---------------------------------------------------------------- ID: e871f741-7dd6-49c6-9b4a-af0d68e647e7 Description: DB Home Patching: Home Id is 86b6a068-55a8-4171-9f94-48b86f135065 Status: Success Created: May 23, 2018 1:37:00 PM CEST Message: WARNING::Failed to run datapatch on db DB01TST##WARNING::Failed to run datapatch on db DB03TST Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- clusterware patch verification May 23, 2018 1:37:21 PM CEST May 23, 2018 1:37:23 PM CEST Success Patch location validation May 23, 2018 1:37:23 PM CEST May 23, 2018 1:37:23 PM CEST Success Opatch updation May 23, 2018 1:37:23 PM CEST May 23, 2018 1:37:23 PM CEST Success Patch conflict check May 23, 2018 1:37:23 PM CEST May 23, 2018 1:37:23 PM CEST Success db upgrade May 23, 2018 1:37:23 PM CEST May 23, 2018 1:37:23 PM CEST Success SqlPatch upgrade May 23, 2018 1:37:23 PM CEST May 23, 2018 1:37:57 PM CEST Success SqlPatch upgrade May 23, 2018 1:37:57 PM CEST May 23, 2018 1:38:26 PM CEST Success SqlPatch upgrade May 23, 2018 1:38:26 PM CEST May 23, 2018 1:38:51 PM CEST Success SqlPatch upgrade May 23, 2018 1:38:51 PM CEST May 23, 2018 1:39:49 PM CEST Success SqlPatch upgrade May 23, 2018 1:39:49 PM CEST May 23, 2018 1:40:28 PM CEST Success SqlPatch upgrade May 23, 2018 1:40:28 PM CEST May 23, 2018 1:40:53 PM CEST Success SqlPatch upgrade May 23, 2018 1:40:53 PM CEST May 23, 2018 1:41:19 PM CEST Success SqlPatch upgrade May 23, 2018 1:41:19 PM CEST May 23, 2018 1:41:44 PM CEST Success SqlPatch upgrade May 23, 2018 1:41:44 PM CEST May 23, 2018 1:42:14 PM CEST Success SqlPatch upgrade May 23, 2018 1:42:14 PM CEST May 23, 2018 1:42:48 PM CEST Success SqlPatch upgrade May 23, 2018 1:42:48 PM CEST May 23, 2018 1:43:22 PM CEST Success
It’s longer but less than 10 minutes for this example. You can see the number of databases here (one line SqlPatch upgrade for one database). The job is successful, but actually not so successful because 2 warnings are raised. And be careful because message field in the job details is limited: check the /opt/oracle/dcs/log/dcs-agent.log for extended warning messages.
Always check the DB Version after the update of all dbhomes, this is the new version of the binaries:
odacli list-dbhomes ID Name DB Version Home Location Status ---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ---------- 86b6a068-55a8-4171-9f94-48b86f135065 OraDB12102_home1 12.1.0.2.180116 (26925218, 26925263) /u01/app/oracle/product/12.1.0.2/dbhome_1 Configured c6e7d556-5785-41d8-a120-ed3ef756848a OraDB11204_home1 11.2.0.4.180116 (26609929, 26925576) /u01/app/oracle/product/11.2.0.4/dbhome_1 Configured
If update job is really successful, jump to step 8.
7) Update the databases where datapatch failed
For all the databases where datapatch cannot be applied, you’ll have to do it manually. I discovered that if your database is not in AMERICAN_AMERICA language and territory (FRENCH_FRANCE in my case) odacli will not be able to apply the datapatch on the database. Let’s check the status of the datapatch on the database, set the NLS_LANG environment to AMERICAN_AMERICA and apply the patch on these databases:
su – oracle . oraenv <<< DB01TST cd $ORACLE_HOME/OPatch sqlplus / as sysdba SQL> set serveroutput on SQL> exec dbms_qopatch.get_sqlpatch_status; Patch Id : 26609798 Action : APPLY Action Time : 24-SEP-2017 23:50:14 Description : DATABASE BUNDLE PATCH 12.1.0.2.170814 Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/26609798/21481992/26609798_apply_G100316_CD BROOT_2017Sep24_23_48_22.log Status : SUCCESS Patch Id : 26925263 Action : APPLY Action Time : 29-MAY-2018 15:40:27 Description : DATABASE BUNDLE PATCH 12.1.0.2.180116 Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/26925263/21857460/26925263_apply_DB01TST_20 18May29_15_40_13.log Status : FAILED PL/SQL procedure successfully completed. exit export NLS_LANG=AMERICAN_AMERICA.AL32UTF8 ./datapatch -verbose sqlplus / as sysdba SQL> set serveroutput on SQL> exec dbms_qopatch.get_sqlpatch_status; Patch Id : 26609798 Action : APPLY Action Time : 24-SEP-2017 23:50:14 Description : DATABASE BUNDLE PATCH 12.1.0.2.170814 Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/26609798/21481992/26609798_apply_G100316_CD BROOT_2017Sep24_23_48_22.log Status : SUCCESS Patch Id : 26925263 Action : APPLY Action Time : 29-MAY-2018 15:40:27 Description : DATABASE BUNDLE PATCH 12.1.0.2.180116 Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/26925263/21857460/26925263_apply_DB01TST_20 18May29_15_40_13.log Status : FAILED Patch Id : 26925263 Action : APPLY Action Time : 29-MAY-2018 15:53:43 Description : DATABASE BUNDLE PATCH 12.1.0.2.180116 Logfile : /u01/app/oracle/cfgtoollogs/sqlpatch/26925263/21857460/26925263_apply_DB01TST_20 18May29_15_53_01.log Status : SUCCESS exit
Repeat this operation on all the faulty databases.
8) Patching the storage
Patching the storage is actually patching the NVMe disks. This is not always mandatory as your disks can already have the latest version. So, if needed, apply the patch:
odacli update-storage -v 12.2.1.3.0 { "jobId" : "4a221df4-8c85-4f34-aa7f-e014cdb751f7", "status" : "Created", "message" : "Success of Storage Update may trigger reboot of node after 4-5 minutes. Please wait till node restart", "reports" : [ ], "createTimestamp" : "May 29, 2018 13:27:41 PM CEST", "resourceList" : [ ], "description" : "Storage Firmware Patching", "updatedTime" : "May 29, 2018 13:27:41 PM CEST" } odacli describe-job -i "4a221df4-8c85-4f34-aa7f-e014cdb751f7" Job details ---------------------------------------------------------------- ID: 4a221df4-8c85-4f34-aa7f-e014cdb751f7 Description: Storage Firmware Patching Status: Success Created: May 29, 2018 1:27:41 PM CEST Message: Task Name Start Time End Time Status ---------------------------------------- ----------------------------------- ----------------------------------- ---------- Applying Firmware Disk Patches May 29, 2018 1:27:41 PM CEST May 29, 2018 1:27:45 PM CEST Success Applying Firmware Expander Patches May 29, 2018 1:27:45 PM CEST May 29, 2018 1:27:52 PM CEST Success Applying Firmware Controller Patches May 29, 2018 1:27:52 PM CEST May 29, 2018 1:28:51 PM CEST Success preRebootNode Actions May 29, 2018 1:28:51 PM CEST May 29, 2018 1:28:51 PM CEST Success Reboot Ilom May 29, 2018 1:28:51 PM CEST May 29, 2018 1:28:51 PM CEST Success
Beware of the reboot of the server few minutes after the patching is finished! And control the components version after the reboot.
9) Optional: deploy the latest db clone files
If you’ll never deploy a new dbhome this step is not necessary. If you will, or simply if you don’t know if you’ll have to do that later, download and register the new db clone files in the repository to be able to create a new dbhome at the same patch level, for example:
cd /opt/patch unzip p23494992_122130_Linux-x86-64.zip Archive: p23494992_122130_Linux-x86-64.zip extracting: odacli-dcs-12.2.1.3.0-180320-DB-12.1.0.2.zip inflating: README.txt update-image --image-files odacli-dcs-12.2.1.3.0-180320-DB-12.1.0.2.zip Unpacking will take some time, Please wait... Unzipping odacli-dcs-12.2.1.3.0-180320-DB-12.1.0.2.zip
10) Control the version of the components
Now the patching is done. It took about 2 hours, not so bad for all these updates. Finally, do a last check of the components to see if everything is fine:
odacli describe-component System Version --------------- 12.2.1.3.0 Component Installed Version Available Version ---------------------------------------- -------------------- -------------------- OAK 12.2.1.3.0 up-to-date GI 12.2.0.1.180116 up-to-date DB { [ OraDB12102_home1 ] 12.1.0.2.180116 up-to-date [ OraDB11204_home1 ] 11.2.0.4.180116 up-to-date } DCSAGENT 18.1.3.0.0 up-to-date ILOM 4.0.0.24.r121140 up-to-date BIOS 39090000 up-to-date OS 6.9 up-to-date FIRMWARECONTROLLER { [ c6 ] 4.650.00-7176 up-to-date [ c0,c1,c2,c3,c4,c5 ] KPYAGR3Q up-to-date } FIRMWAREDISK 0R3Q up-to-date
Everything should be up-to-date now, until the next patch 🙂