Yesterday I had fun with Opatch again. We wanted to apply the latest PSU (12.1.0.2.170418) on top of a 12.1.0.2 GI without any patches applied:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
xxxxx@xxxxx:/disk00/app/12.1.0/grid_2_4/diagnostics/ [+ASM2] opatch lsinv
Oracle Interim Patch Installer version 12.2.0.1.8
Copyright (c) 2017, Oracle Corporation. All rights reserved.
 
Oracle Home : /disk00/app/12.1.0/grid_2_4
Central Inventory : /disk00/app/oraInventory
from : /disk00/app/12.1.0/grid_2_4/oraInst.loc
OPatch version : 12.2.0.1.8
OUI version : 12.1.0.2.0
Log file location : /disk00/app/12.1.0/grid_2_4/cfgtoollogs/opatch/opatch2017-05-17_15-44-20PM_1.log
 
Lsinventory Output file location : /disk00/app/12.1.0/grid_2_4/cfgtoollogs/opatch/lsinv/lsinventory2017-05-17_15-44-20PM.txt
 
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dbprd08s.psi.ch
ARU platform id: 226
ARU platform description:: Linux x86-64
 
Installed Top-level Products (1):
 
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
 
 
There are no Interim patches installed in this Oracle Home.
 
 
Patch level status of Cluster nodes :
 
Patching Level Nodes
-------------- -----
0 xxxxx,xxxxx
 
--------------------------------------------------------------------------------
 
OPatch succeeded.

As you can see from the above output we applied the latest Opatch version for 12.1.0.2, which is this one:
Selection_001

When we kicked off opatchauto:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
     
Customer Problem Description
---------------------------------------------------
 
Problem Summary
---------------------------------------------------
opatchauto - Topology creation failed.
 
Problem Description
---------------------------------------------------
[root@xxxxx patchset]$ opatchauto apply /disk00/app/oracle/patchset/25434003 -oh /disk00/app/12.1.0/grid_2_4 -ocmrf /home/cluster_admin/ocm.rsp
 
OPatchauto session is initiated at Wed May 17 15:30:20 2017
 
System initialization log file is /disk00/app/12.1.0/grid_2_4/cfgtoollogs/opatchautodb/systemconfig2017-05-17_03-30-22PM.log.
 
Exception in thread "main" java.lang.UnsatisfiedLinkError: oracle.cluster.impl.common.cnative.ClusterNative.CLSBInit(ILjava/lang/String;ZLoracle/cluster/impl/common/cnative/ClusterNativeResult;Ljava/lang/Object;)J
at oracle.cluster.impl.common.cnative.ClusterNative.CLSBInit(Native Method)
at oracle.cluster.util.SRVMContext.init(SRVMContext.java:130)
at oracle.cluster.util.SRVMContext.init(SRVMContext.java:83)
at oracle.ops.mgmt.has.HASContext.(HASContext.java:135)
at oracle.ops.mgmt.has.HASContext.getInstance(HASContext.java:292)
at oracle.ops.mgmt.has.ClusterUtil.(ClusterUtil.java:80)
at oracle.cluster.impl.common.ClusterImpl.(ClusterImpl.java:114)
at oracle.cluster.impl.common.CommonFactoryImpl.getCluster(CommonFactoryImpl.java:81)
at oracle.cluster.common.CommonFactory.getCluster(CommonFactory.java:79)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.getSrvmCluster(CrsProductDriver.java:506)
at com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver.buildSystemInstance(CrsProductDriver.java:187)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator.createSystemInstance(TopologyCreator.java:204)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator.process(TopologyCreator.java:142)
at com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.TopologyCreator.main(TopologyCreator.java:99)
 
OPatchauto session completed at Wed May 17 15:30:23 2017
Time taken to complete the session 0 minute, 3 seconds
 
Topology creation failed.

Oh, surprise. We did exactly the same one day before on another Cluster and all went fine. When you check what MOS suggest for this you will not find much information (yes, this is the 64bit version of Opatch). The log file is not much of a help either:

1
2
3
4
5
6
cluster_admin@xxxxx:/home/cluster_admin/ [+ASM2] cat /disk00/app/12.1.0/grid_2_4/cfgtoollogs/opatchautodb/systemconfig2017-05-17_03-30-22PM.log
2017-05-17 15:30:22,800 INFO [1] com.oracle.glcm.patch.auto.db.product.inventory.ClusterInformationLoader - crsType: CRS
2017-05-17 15:30:22,960 INFO [1] com.oracle.glcm.patch.auto.db.product.inventory.ClusterInformationLoader - running: true
2017-05-17 15:30:22,979 INFO [1] com.oracle.glcm.patch.auto.db.product.inventory.ClusterInformationLoader - running: true
2017-05-17 15:30:23,199 INFO [1] com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver - crsType: CRS
2017-05-17 15:30:23,350 INFO [1] com.oracle.glcm.patch.auto.db.product.driver.crs.CrsProductDriver - running: true

Five lines and all seems to be fine. The only obvious difference to what worked the day before is that we did not have any patch applied to the GI yet. Once we downgraded Opatch to 12.1.0.1.8 all went fine:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@xxxxxx grid_2_4]$ opatchauto apply /disk00/app/oracle/patchset/25434003 -oh /disk00/app/12.1.0/grid_2_4 -ocmrf /home/cluster_admin/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
 
OPatchauto Version : 12.1.0.1.8
OUI Version : 12.1.0.2.0
Running from : /disk00/app/12.1.0/grid_2_4
 
opatchauto log file: /disk00/app/12.1.0/grid_2_4/cfgtoollogs/opatchauto/25434003/opatch_gi_2017-05-17_16-34-00_deploy.log
 
Parameter Validation: Successful
 
Configuration Validation: Successful
...

I could not verify this on my local lab so I am not sure what exactly caused this, but at least the workaround did the trick.

Hope this helps in case you run into the same issue.