Patching Oracle Database Appliance From 18.8 to 19.6

By Mouhamadou Diaw

The ODA software 19.6 is released and people are starting to patch.
A direct patch to version 19.6 is possible from version 18.8.
Before patching your deployment to Oracle Database Appliance release 19.6, you must upgrade the operating system to Oracle Linux 7.
In this blog I am describing the steps I follow when patching an ODA from 18.8 to 19.6. I am using an ODA X7-2 (one node)
The first step is of course to download the required patch. You will need to copy following patch to your ODA.
Patch 31010832 : ORACLE DATABASE APPLIANCE 19.6.0.0.0 SERVER PATCH FOR ALL ODACLI/DCS STACK
Patch 30403662 : ORACLE DATABASE APPLIANCE RDBMS CLONE FOR ODACLI/DCS STACK

The first patch 31010832 will be used to upgrade the OS to Linux 7 and to patch you server to Appliance 19.6
The patch 30403662 will install the Oracle 19c clone database

The first patch contains 4 files

Just use the unzip command to decompress

Once done we have following files that we will use to update the repository

After you will have to list your scheduled jobs with the list-schedules option

And disable all enabled jobs (column disabled to false). This done by using the command update-server with -d for disable and -i for the job_id

When listing the jobs again, the column disabled must return true for all jobs

odabr is required. for the patching to 19.6. odabr is a tool to backup and recover an ODA. When running the precheck. the result will return failed if odabr is not installed. Please you can consult following document for downloading and using odabr : ODA (Oracle Database Appliance): ODABR a System Backup/Restore Utility (Doc ID 2466177.1)
By default odabr requires 190G free space in the LVM which can be not the case sometimes. In such case odabr should be run using some specific options. In my case I used the command below to take a backup of ODA.

Snapshots can be verified using the infosnap option

We can now update the repository with the server software using the patch 31010832

For the first file

Check the job status with describe-job

Do this for all 3 other files. All jobs must return success.

Now let’s update the DCS agent

Confirm that the job returns success

Before upgrading the OS, we must generate a prepatch report. This will indicate eventual issues that may cause the patching to fail.

We can monitor the status of the job with the describe-job command. As we can see we have failure

To have more info about these errors, we use the describe-prepatchreport

As we can see the prepatch reports two errors
One related to odabr because I have already taken a backup for my ODA
A second error related to custom rpms installed.

For the error related to odabr as it is mentioned in the documentation

If snapshots are already present on the system when odacli create-prepatchreport is run, this precheck fails, because ODACLI expects to create these snapshots itself. If the user created snapshots or the operating system upgrade was retried (due to a failure) after it had already created the snapshots, this precheck will fail. Note that if snapshots already exist, odacli update-server –c OS still continues with the upgrade.

I decided to ignore the error and to run later the patch with option –force

But be sure before continuing to check these following two files and to remove all mentioned rpm packages.

Just another thing os to verify if you have enough free space. You can if needed cleanup your old repository

In my case below is the status of my /, /u01 and /opt filesystems.

It’s time now to start the patching process by upgrading the server to Linux 7. I did the following command from the ILOM console as described in the documentation.

With putty connect to your ILOM IP with root. Once connect to the ILOM just type start /SP/console and log in as root on your server.

And with root from the ILOM console run the OS upgrade (Just remember in my case I decided to use the –force)

This will take some time, but just wait. You may see the progression from the ILOM console

At the end of the process the server will reboot. And if everything is fine you should have an Linux 7.7 now

After the operating system upgrade is completed successfully, run the post upgrade checks:

As specified let’s remove the snapshots we took with odabr

Running the posthecks again

After the OS upgrade now let’s upgrade the remaining components
We start by updating the DSC agent

We validate that the Job returns success

We update the DCS components

And then we update the server

This will lasts about 45 minutes. Validate that all is fine

And we can verify that the components were upgraded

After the server we have to update the storage

We can verify that the job was successful

I did not patch the existing database home because I just want to keep the actual version for my existing databases.

To be able to create new 19c databases we have to update the repository with corresponding rdbms clone.
After unzipping the archive

I just have to update the repository with the db clone

Then I check that the job was fine

Encountered ACFS Issue
During the patch I did not have any issue. All was fine. But at the end of the patch my existing databases did not come up because of an acfs issue.
Indeed, the ASM instance was up but the ASM proxy instance APX was instable

Looking on the trace file

For some reason the acfs module was not loaded. My databases was Oracle 11.2.0.4 and were using acfs.

So I first stop my crs

And then I reinstall the acfs module with following acfsroot command

Running again the lsmod command, I see that things are better

After I reboot my server and all databases and the ASM Proxy Instance become up and running. I just wanted to mention this issue I had and hope this will help.

Conclusion

Some recommendations
Remove all rpm packages that were manually installed
Verify that you have enough space on / , /u01 and /opt filesystems. /opt and /u01 can be increased online
Launch the OS upgrade from the ILOM console
I hope this blog will help.