Introduction

Patching an Oracle Database Appliance takes at least 3-4 hours to complete. Considering that all components are being patched together (OS, GI, ILOM, BIOS, DB homes, firmwares, databases), it’s not that long. You should only consider these 3-4 hours of work if everything is prepared prior applying the patch. Preparation will need additional work and therefore time. And in specific cases, applying the patch could take longer than expected. This is what could eventually happen regarding ASR Manager update.

Time needed for applying a patch

It’s quite tough to define an accurate time for patching, at least for your first ODA. Actually, patching the first one will define the time needed for patching the others. For sure, this is only true if your ODAs are identical and at the same patch level.

Here are the key factors that could impact the patching time:

  • preparation: downloading the patch files, copying the files to the server, cleaning up the filesystems has to be done before
  • procedure: patching is a multi-step process involving quite a lot of commands, having a good procedure is better than looking into the documentation and into MOS when troubleshooting. Most often you will issue the patching procedure after you successfully applied the patch on the first ODA
  • ODA model: HA ODAs have 2 nodes, therefore you will need more time to patch them. ODAs with spinning disks will also require more time to patch. ODAs with spinning disks are for example the old but still supported X5-2HA or those recent HA ODAs in High-Capacity flavor
  • patch content: some patches are limited to GI and DB updates, and are easier to apply. Major OS upgrades, or a lot of components to upgrade will make the patch take more time to complete
  • system cleanness: the cleanest your system is, the less troubleshooting you will need
  • number of DB homes and databases: patch is applied at the DB home level as well as at database level, therefore a lot of databases and DB homes means longer patching
  • use of DB Systems: when using virtualization, each database will have its own VM when they are configured as DB Systems. A complete patching of your ODA will imply patching each DB System, meaning several hours more of patching time
  • starting version: patch are not always cumulative. If you start from a very old version, you will need to apply multiple patches, and multiply the time needed for patching by the number of patches you’ll have to apply

Patching steps

Patching is divided in several steps. After registering the patch file on your ODA, you will have to:

  • update the DCS binaries: this involves 3 tasks
  • update the server: this part is the most complex one because it includes OS, ILOM, BIOS, local firmwares and GI patches
  • update the storage: this is only related to controllers and data disks firmwares
  • update the dbhomes and the databases: this has to be done on each installed DB home if you want to bring your databases to the latest patch level

You won’t go through these tasks without any backup and control: a backup of the system is highly recommended before starting the patching process (with ODABR for example), and each update will need running a precheck before to make sure your components are ready to be patched.

Applying multiple patches means repeating all these steps for the next ones.

Problem with ASR Manager RPM update

ASR Manager RPM update is one of the multiple subtasks of the odacli update-server. This is not something critical and/or complex, and it usually lasts less than 10 minutes. It’s been 2 times, on 2 different ODAs, that this update hangs the update-server. This recently happened on a X8-2S ODA jumping from 19.7 to 19.10 (target version was 19.18 and required applying 3 patches).

When querying the update-server job, it’s stuck for minutes or eventually hours on this subtask:

odacli describe-job -i "23852ffd-46da-4b32-96c8-4ab08bda3109"

Job details
----------------------------------------------------------------
                     ID:  23852ffd-46da-4b32-96c8-4ab08bda3109
            Description:  Server Patching
                 Status:  Running
                Created:  March 30, 2023 9:19:49 AM CEST
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Patch location validation                March 30, 2023 9:19:57 AM CEST      March 30, 2023 9:19:57 AM CEST      Success
dcs-controller upgrade                   March 30, 2023 9:19:57 AM CEST      March 30, 2023 9:19:57 AM CEST      Success
Creating repositories using yum          March 30, 2023 9:19:57 AM CEST      March 30, 2023 9:19:59 AM CEST      Success
Updating YumPluginVersionLock rpm        March 30, 2023 9:20:00 AM CEST      March 30, 2023 9:20:00 AM CEST      Success
Applying OS Patches                      March 30, 2023 9:20:00 AM CEST      March 30, 2023 9:29:48 AM CEST      Success
Creating repositories using yum          March 30, 2023 9:29:48 AM CEST      March 30, 2023 9:29:48 AM CEST      Success
Applying HMP Patches                     March 30, 2023 9:29:48 AM CEST      March 30, 2023 9:30:04 AM CEST      Success
Client root Set up                       March 30, 2023 9:30:04 AM CEST      March 30, 2023 9:30:07 AM CEST      Success
Client grid Set up                       March 30, 2023 9:30:07 AM CEST      March 30, 2023 9:30:10 AM CEST      Success
Patch location validation                March 30, 2023 9:30:10 AM CEST      March 30, 2023 9:30:10 AM CEST      Success
oda-hw-mgmt upgrade                      March 30, 2023 9:30:10 AM CEST      March 30, 2023 9:30:40 AM CEST      Success
OSS Patching                             March 30, 2023 9:30:40 AM CEST      March 30, 2023 9:30:41 AM CEST      Success
Applying Firmware Disk Patches           March 30, 2023 9:30:42 AM CEST      March 30, 2023 9:30:52 AM CEST      Success
Applying Firmware Controller Patches     March 30, 2023 9:30:52 AM CEST      March 30, 2023 9:32:05 AM CEST      Success
Checking Ilom patch Version              March 30, 2023 9:32:07 AM CEST      March 30, 2023 9:32:08 AM CEST      Success
Patch location validation                March 30, 2023 9:32:08 AM CEST      March 30, 2023 9:32:09 AM CEST      Success
Save password in Wallet                  March 30, 2023 9:32:11 AM CEST      March 30, 2023 9:32:11 AM CEST      Success
Apply Ilom patch                         March 30, 2023 9:32:11 AM CEST      March 30, 2023 9:40:45 AM CEST      Success
Copying Flash Bios to Temp location      March 30, 2023 9:40:45 AM CEST      March 30, 2023 9:40:45 AM CEST      Success
Patch location validation                March 30, 2023 9:40:45 AM CEST      March 30, 2023 9:40:45 AM CEST      Success
ASR Manager RPM update                   March 30, 2023 9:40:45 AM CEST      March 30, 2023 9:40:45 AM CEST      Running

Don’t waste your time waiting too long, it will never finish. The reason is that you may have old ASR processes running for months or eventually years on your system. Let’s have a look at the ASR processes:

ps -ef | grep asr

root     31894     1 99  2020 ?        7986-06:19:24 /usr/java/jdk1.8.0_241-amd64/jre/bin/java -Xms512m -Xmx1536m -Dlog4j.configurationFile=/opt/asrmanager/configuration/log4j2.xml -Dasr.log.level=info -Dasr.log.filecount=5 -Dauditlog.days=30 -cp /opt/asrmanager/felix-framework/bin/felix.jar org.apache.felix.main.Main /opt/asrmanager/bundle-cache/ -b /opt/asrmanager/lib/asrstart
root     59341 84372  0 09:40 ?        00:00:00 /bin/sh /opt/asrmanager/bin/asr stop
root     59388 59341  0 09:40 ?        00:00:00 /bin/sh /sbin/service asrm stop
root     59393 59388  0 09:40 ?        00:00:00 /bin/sh /etc/init.d/asrm stop
root     59394 59393  0 09:40 ?        00:00:00 /bin/sh /opt/asrmanager/bin/asrm stop
root     59495 59394  0 09:40 ?        00:00:00 /bin/sh /opt/asrmanager/bin/asr halt
root     59643 59495  0 09:40 ?        00:00:00 /usr/java/jdk1.8.0_241-amd64/jre/bin/java -cp /opt/asrmanager/lib/com.oracle.asr.container.jar:/opt/asrmanager/lib/commons-net-1.4.1.jar com.oracle.asr.container.cli.AsrCommand halt
root     82964     1  0 08:53 ?        00:00:00 /bin/sh /opt/asrmanager/bin/asr restart
root     83037 82964  0 08:53 ?        00:00:00 /bin/sh /sbin/service asrm restart
root     83044 83037  0 08:53 ?        00:00:00 /bin/sh /etc/init.d/asrm restart
root     83049 83044  0 08:53 ?        00:00:00 /bin/sh /etc/init.d/asrm stop
root     83051 83049  0 08:53 ?        00:00:00 /bin/sh /opt/asrmanager/bin/asrm stop
root     83152 83051  0 08:53 ?        00:00:00 /bin/sh /opt/asrmanager/bin/asr halt
root     83327 83152  0 08:53 ?        00:00:01 /usr/java/jdk1.8.0_241-amd64/jre/bin/java -cp /opt/asrmanager/lib/com.oracle.asr.container.jar:/opt/asrmanager/lib/commons-net-1.4.1.jar com.oracle.asr.container.cli.AsrCommand halt
root     97789 65407  0 10:22 pts/0    00:00:00 grep --color=auto asr

The first one is running since 2020, it could probably be killed without any consequences. Remember that this ODA is currently being patched, so databases are supposed to be bounced.

kill -9 31894

After killing this process, the update-server resumed. But I lost 40 minutes in my patching planning:

odacli describe-job -i "23852ffd-46da-4b32-96c8-4ab08bda3109"
Job details
----------------------------------------------------------------
                     ID:  23852ffd-46da-4b32-96c8-4ab08bda3109
            Description:  Server Patching
                 Status:  Failure
                Created:  March 30, 2023 9:19:49 AM CEST
                Message:  DCS-10001:Internal error encountered:  apply patch using OpatchAuto on node chlaora02.

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Server patching                          March 30, 2023 9:19:56 AM CEST      March 30, 2023 10:40:54 AM CEST     Failure
Server patching                          March 30, 2023 9:19:56 AM CEST      March 30, 2023 10:40:54 AM CEST     Failure
Patch location validation                March 30, 2023 9:19:57 AM CEST      March 30, 2023 9:19:57 AM CEST      Success
dcs-controller upgrade                   March 30, 2023 9:19:57 AM CEST      March 30, 2023 9:19:57 AM CEST      Success
Creating repositories using yum          March 30, 2023 9:19:57 AM CEST      March 30, 2023 9:19:59 AM CEST      Success
Updating YumPluginVersionLock rpm        March 30, 2023 9:20:00 AM CEST      March 30, 2023 9:20:00 AM CEST      Success
Applying OS Patches                      March 30, 2023 9:20:00 AM CEST      March 30, 2023 9:29:48 AM CEST      Success
Creating repositories using yum          March 30, 2023 9:29:48 AM CEST      March 30, 2023 9:29:48 AM CEST      Success
Applying HMP Patches                     March 30, 2023 9:29:48 AM CEST      March 30, 2023 9:30:04 AM CEST      Success
Client root Set up                       March 30, 2023 9:30:04 AM CEST      March 30, 2023 9:30:07 AM CEST      Success
Client grid Set up                       March 30, 2023 9:30:07 AM CEST      March 30, 2023 9:30:10 AM CEST      Success
Patch location validation                March 30, 2023 9:30:10 AM CEST      March 30, 2023 9:30:10 AM CEST      Success
oda-hw-mgmt upgrade                      March 30, 2023 9:30:10 AM CEST      March 30, 2023 9:30:40 AM CEST      Success
OSS Patching                             March 30, 2023 9:30:40 AM CEST      March 30, 2023 9:30:41 AM CEST      Success
Applying Firmware Disk Patches           March 30, 2023 9:30:42 AM CEST      March 30, 2023 9:30:52 AM CEST      Success
Applying Firmware Controller Patches     March 30, 2023 9:30:52 AM CEST      March 30, 2023 9:32:05 AM CEST      Success
Checking Ilom patch Version              March 30, 2023 9:32:07 AM CEST      March 30, 2023 9:32:08 AM CEST      Success
Patch location validation                March 30, 2023 9:32:08 AM CEST      March 30, 2023 9:32:09 AM CEST      Success
Save password in Wallet                  March 30, 2023 9:32:11 AM CEST      March 30, 2023 9:32:11 AM CEST      Success
Apply Ilom patch                         March 30, 2023 9:32:11 AM CEST      March 30, 2023 9:40:45 AM CEST      Success
Copying Flash Bios to Temp location      March 30, 2023 9:40:45 AM CEST      March 30, 2023 9:40:45 AM CEST      Success
Patch location validation                March 30, 2023 9:40:45 AM CEST      March 30, 2023 9:40:45 AM CEST      Success
ASR Manager RPM update                   March 30, 2023 9:40:45 AM CEST      March 30, 2023 10:29:01 AM CEST     Success
Modify JavaExec Path                     March 30, 2023 10:29:01 AM CEST     March 30, 2023 10:29:01 AM CEST     Success
Remove AsrConfBackup File                March 30, 2023 10:29:01 AM CEST     March 30, 2023 10:29:01 AM CEST     Success
Starting the clusterware                 March 30, 2023 10:29:01 AM CEST     March 30, 2023 10:30:34 AM CEST     Success
...

Conclusion

I would definitely recommend doing a reboot of your ODA nodes prior patching. And monitor the patching subtasks quite closely. Normally, subtasks are chained and should sooner or later end with a Success or a Failure, but some of them may be stuck until you do something.

Post Views: 1,538
Thumbnail [60x60]
by
Jérôme Dubar