Recovering from failed patch on virtualized ODA

When a patch fails on a virtualized Oracle Database Appliance (ODA), this ODA is often unusuable because Linux and OAKD are patched to new release but Grid Infrastructure is still on old version. OAKD cannot be restarted in default mode because in this mode the active Grid Infrastructure version is checked, which will fail due to old version. Also Grid Infrastructure cannot be started due to the fact that OAKD controls access of shared hardware on ODA and if OAKD does not run, shared hardware cannot be accessed.

One way to resolve this problem is to reimage the ODA, which is time consuming and means that all databases and VMs have to be restored.

A workaround of this chicken and egg problem (I cannot guarantee that it is supported) as a last try before reimaging the ODA could be to start OAKD in non-cluster mode. This not very good documented mode does not check active grid infrastructure but gives access to shared hardware. Additional VMs cannot be started because there is no master OAKD. In this mode manual patching/upgrade of Grid Infrastructure is possible.

The non cluster mode can be entered like following (on every ODA node):


cp /opt/oracle/oak/install/oakdrun /opt/oracle/oak/install/oakdrun_orig
echo "non-cluster" > /opt/oracle/oak/install/oakdrun
cd /etc/init.d
./init.oak start
…
[root@xx init.d]# ps -ef | grep oakd
root 49697 49658 11 11:05 ? 00:00:02 /opt/oracle/oak/bin/oakd -non-cluster
root 50511 42821 0 11:05 pts/0 00:00:00 grep oakd


Now Grid Infrastructure patching or upgrade can be done.

If only an ODA_BASE VM exists and timeframe for manual patching/upgrade is too short, it also can be tried is to start Grid Infrastructure on one ODA node and then start the services. Patching or reimaging has to be done in next suitable timeframe.

After running Grid Infrastructure on new version, OAKD can be tried to start in default mode:


echo "start" > /opt/oracle/oak/install/oakdrun
cd /etc/init.d
./init.oak start
…
[root@xx init.d]# ps -ef | grep oakd
root 30187 30117 13 10:18 ? 00:00:02 /opt/oracle/oak/bin/oakd foreground
root 31902 7569 0 10:18 pts/1 00:00:00 grep oakd


Perhaps manual patching/upgrade of other components has to be done afterwards.

After patching/upgrading, ODA has to be checked with:


oakcli show version -detail
oakcli validate -a