GoldenGate 12.2: ERROR: Unable to create a new wallet

Today I had a very strange issue when setting up a brand new GoldenGate 12.2 installation. Installation went fine, patch went fine (PSU 170221) and then I started to configure GoldenGate. One of the first commands we usually do is:

GGSCI (xxxxx) 2> create wallet

I really never had any issues with this, but today I got:

GGSCI (psrdbap03n1) 2> create wallet

ERROR: Unable to create a new wallet.

What a great error message and it was not much more verbose in the GoldenGate logfile:

2017-03-30 15:20:06  INFO    OGG-02096  Oracle GoldenGate Command Interpreter for Oracle:  ERROR: Unable to create a new wallet.

So, what to do? My first thought was:

oracle@xxx]~[xxx]> ps -ef | grep ggscs
oracle    7683 30606  0 15:29 pts/2    00:00:00 grep ggscs
[oracle@xxx]~[xxx]> ps -ef | grep ggs
oracle    1261 24160  0 15:19 pts/0    00:00:00 ./ggsci
oracle    7698 30606  0 15:29 pts/2    00:00:00 grep ggs
[oracle@xxx]~[xxx]> strace -fp 1261

… and then do the command again and check the strace output. But there was not much to look at. GoldenGate tried to stat the wallet files but as they did not exist yet failed to, which is fine:

[pid  1261] stat("dirwlt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid  1261] stat("dirwlt/cwallet.sso", 0x7fff88bd2f30) = -1 ENOENT (No such file or directory)
[pid  1261] stat("dirwlt/cwallet.lck", 0x7fff88bd28f0) = -1 ENOENT (No such file or directory)
[pid  1261] open("dirwlt/cwallet.lck", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 6
[pid  1261] close(6)                    = 0
[pid  1261] open("dirwlt/cwallet.lck", O_RDWR) = 6

Some lines later:

[pid  1262] futex(0x1e68a64, FUTEX_WAIT_PRIVATE, 123, {0, 982142} 
[pid  1261] write(1, "ERROR: Unable to create a new wa"..., 37) = 37
[pid  1261] write(1, "\n", 1)    

And then many, many of these:

[pid  1262] futex(0x1e68a64, FUTEX_WAIT_PRIVATE, 163, {0, 63977012}) = -1 ETIMEDOUT (Connection timed out)
[pid  1262] futex(0x1e68a38, FUTEX_WAKE_PRIVATE, 1) = 0

In short, I had no clue what was the issue (the many futex calls did not help me) . The next idea was to enable debugging for Goldengate. This can be done by creating a file like this in the GoldenGate Home (a picture again, sorry, the xml tags destroy the formatting. Let me know when you want to have the file):

This will create a file called “gglog-ggsci.log” in the GoldenGate Home which contains more information what was going on. But the only two lines which showed “ERROR” were these:

2017-03-30 16:09:31.015 ERROR|ggsec.CWallet                 |main| 1090 ggsec/CWallet.cpp        | initNZCtx: Error initializing the nz context. NZERROR 43490
2017-03-30 16:09:31.015 ERROR|ggsec.CWallet                 |main|  439 ggsec/CWallet.cpp        | create: Could not initialize network security context.

Hm, this does not help me either (mos did not come up with a single note about this error). To make it short: Finally this happened because I had the ORACLE_HOME environment set to the Goldengate Home:

[oracle@xxxxxxx]/u01/app/ogg/ch_src/product/12.2.0.1.170221/via[p1viach1]> echo $ORACLE_HOME
/u01/app/ogg/ch_src/product/12.2.0.1.170221/via

As soon as I changed this to a database home everything was fine again:

[oracle@xxx]/u01/app/ogg/ch_src/product/12.2.0.1.170221/via[p1viach1]> export ORACLE_HOME=/u01/app/oracle/product/11.2.0.3/dbhome_1/
[oracle@xxx]/u01/app/ogg/ch_src/product/12.2.0.1.170221/via[p1viach1]> ./ggsci 

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.170221 OGGCORE_12.2.0.1.0OGGBP_PLATFORMS_170123.1033_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Jan 23 2017 21:54:15
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.

GGSCI (xxx) 1> create wallet

Created wallet at location 'dirwlt'.

Opened wallet at location 'dirwlt'.

GGSCI (xxx) 2> 

When you ask yourself why I had the ORACLE_HOME pointed to the GoldenGate Home: I applied the latest PSU just before and before invoking opatch I did set the ORACLE_HOME to point to the GoldenGate Home. When you do the same in GoldenGate 12.1 and even in the GoldenGate 12.2 base release you will not hit this. Currently there is bug open with Oracle, lets see what happens 🙂